An AI coding agent can produce a lot of visible activity. It can open files, draft tests, write code, summarize a diff, and propose several approaches before an engineer has finished a cup of coffee. That makes it tempting to measure the activity: prompts per engineer, lines changed, suggestions accepted, pull requests opened, or agent hours consumed.
Those are easy numbers to collect. They are also a reliable way to teach a team to optimize for theater.
Engineering productivity is not the amount of keyboard or agent activity a team can observe. It is the ability to turn worthwhile problems into safe, useful outcomes with reasonable speed and sustainable effort. AI changes the tools inside that system. It does not change the system's purpose. A good measurement approach should help a team find friction, test an improvement, and decide whether the improvement was worth keeping. It should not turn individual developers into telemetry endpoints.
Start with the decision, not the available data
Before adding a dashboard, name the decision it is meant to improve. “Are we getting more value from AI?” is too broad to produce useful measurement. A better question has an operating boundary:
- Is local verification slow enough that engineers avoid it before opening a pull request?
- Are small maintenance changes waiting too long for review?
- Does an agent-assisted test-writing workflow catch more regressions before merge, or merely create more tests to maintain?
- Are new engineers learning an unfamiliar codebase faster without creating more rework for reviewers?
The decision gives every measure a job. If the team cannot describe what it would do differently after seeing a number, it does not need that number yet. This is particularly important with AI tooling because vendors can expose a great deal of usage data that says more about license consumption than about engineering outcomes.
The right unit of analysis is usually a workflow or a class of work, not a person. A team may compare the time from a well-scoped bug report to a verified fix, the rework required after review, or the reliability of a release path. It does not need a leaderboard of who accepted the most completions.
Measure a balanced system
No single metric captures software delivery. A compact set is more honest when it looks at four related dimensions.
| Dimension | Useful signals | What the signal can reveal |
|---|---|---|
| Flow | Time from ready work to verified change, review wait time, blocked work | Where work sits still or handoffs are expensive |
| Quality | Escaped defects, rollback rate, repeat incidents, review rework | Whether apparent speed moved risk downstream |
| Learning | Time to orient in an unfamiliar area, recurring support requests, confidence in local checks | Whether tools make the system easier to understand and operate |
| Outcome | Customer problem solved, operational burden removed, cost or latency improved | Whether the work was worth doing at all |
These are not a universal scorecard. They are a menu for choosing one or two signals per improvement effort. If a team uses an agent to accelerate a narrow test migration, flow and review rework may be enough. If it is changing how engineers diagnose production failures, the quality and learning signals matter more.
The useful pattern is a before-and-after comparison over enough work to be representative, paired with a short qualitative check. Did reviewers find the changes easier to reason about? Did engineers trust the verification path? Did the new workflow create a new queue somewhere else? Numbers without these questions are often precise but not informative.
Keep attribution modest
AI participates in a workflow; it rarely explains the whole result. A faster pull request may reflect a better test harness, a simpler task, a more familiar service, or an experienced reviewer as much as an agent. Treat an observed change as evidence to investigate, not proof that a tool caused it.
This is a useful discipline for pilots. Select a small class of work, write down the intended improvement, and preserve the conditions that matter: team, repository area, risk level, normal verification steps, and review policy. Run the experiment long enough to see ordinary variation. Then ask whether the team would keep the practice if the tool were no longer novel.
It also means separating tool telemetry from outcome measurement. Usage data can help with capacity planning or support: perhaps a team needs better onboarding, or a repository needs a clearer local command. It should not be used to infer individual effort, competence, or commitment. An engineer who asks an agent for fewer suggestions may simply understand the system well enough to need less exploration.
Do not reward output proxies
Several common AI metrics look appealing because they move quickly. They should be treated as diagnostic context at most, not targets:
- Lines of code generated or accepted. More code is frequently more surface area, not more value.
- Prompts, chats, or agent sessions. These say nothing about whether the resulting decision was good.
- Pull requests per engineer. Splitting work can improve reviewability, but a quota encourages artificial slicing and queue pressure.
- Time spent with the tool. This rewards tool use, not delivery or judgment.
- A single cycle-time average. It can improve because a team stopped taking on difficult work or pushed testing into a later stage.
Every target changes behavior. If leaders publish a weekly “AI adoption” rank, the organization will quickly learn to create the signals being rewarded. That is not a moral failure by developers. It is a predictable response to an incentive system. Leaders own the choice of proxy and the behavior it produces.
Make quality part of the same conversation
Speed is only an improvement if the work remains understandable and safe. Keep the normal engineering controls in the measurement loop: code review, focused tests, reproducible checks, operational signals, and a clear owner for the change. AI-assisted work does not earn a reduced standard because it was fast to produce.
That is why a reviewable handoff matters. How To Design AI-Agent Handoffs That a Human Reviewer Can Actually Trust describes the compact evidence, interpretation, decision, and open-question contract a reviewer needs. How To Keep AI Coding Agent Changes Small Enough To Review explains why a smaller change is a control on uncertainty rather than a style preference.
When a pilot appears to improve flow but increases reviewer rework or incident load, believe the whole system. The tool may still be useful, but the current workflow is not yet an improvement. Perhaps the agent needs a stronger task boundary, a better local test command, or a different class of work.
Use measurement as a learning loop
A practical leadership loop is deliberately small:
- Pick one delivery constraint that is costly enough to matter.
- State the hypothesis: for example, “a shared local verification command will reduce review rework for small agent-assisted fixes.”
- Choose a few balanced signals and a time window.
- Keep the work class and quality bar visible.
- Review the data with the engineers doing the work, including the stories the chart cannot show.
- Keep, adjust, or stop the practice based on the result.
This is closer to an engineering experiment than a performance-management system. It makes disagreement useful. An engineer can say that cycle time fell because the team avoided a hard dependency, or that review quality improved only because one staff engineer absorbed the difficult work. Those observations help the team make a better decision next time.
The AI Engineering hub collects the related practices: bounded agent work, review discipline, reliable verification, and accountable ownership. Measurement belongs beside those practices, not above them as a reporting exercise.
Productivity is a capability, not a surveillance program
The most useful question is not “which developer used AI the most?” It is “did we make it easier for this team to deliver correct work, learn the system, and make sound tradeoffs?” A healthy answer might include a faster feedback loop, a clearer handoff, fewer repeat failures, or more time for the work that requires human judgment.
That answer will rarely fit in a single number. It should not. Engineering is a system of people, tools, constraints, and consequences. Measure it with enough humility to improve the system, and enough discipline not to mistake observable activity for progress.
