James Hobbs Discuss an engagement

How do you measure whether AI coding tools improve engineering delivery?

Short answer

Measure AI coding tools the way you measure delivery: take a baseline of lead time, review effort, change failure rate and defects, run a time-boxed pilot with one team, and compare. Lines of code and acceptance rates are poor signals. The question is whether customers get working software sooner, with quality held.

Which measures matter?

MeasureWhat it tells youWatch out for
Lead time for changesWhether work reaches customers fasterGains lost in review queues
Review effortWhether reviewers are overloaded by volumeRubber-stamping large AI diffs
Change failure rateWhether quality holdsRising incidents after release
Defects and reworkWhether code needs fixing laterDelayed effects over months
Size of changesWhether diffs stay reviewableCode bloat and duplication

Running a fair pilot

  1. Take two to four weeks of baseline measures before changing anything
  2. Pick one team and agree the workflows, tools and review rules up front
  3. Add automated checks so AI-written code meets the same bar as human code
  4. Compare after four to six weeks, and decide whether to roll out

What I put in place

In teams I work with, the gains come from the system around the tools, not the tools alone:

  • Automated code review with CodeRabbit and Codex, so people spend review time on what matters
  • AGENTS.md files in each repository, kept up to date and shared, so coding agents follow the team’s conventions
  • A shared channel where engineers post what works and what does not
  • CI/CD scaled for the volume of code AI now produces, with its runtime tracked and improved continually

I code almost every day, so this is not a standard imposed from above. I routinely review how I work, and at least once a week I ask engineers what they are doing differently.

I run many pieces of work in parallel while delivering the core workstream quickly, one step at a time. That relies on developing on remote servers as well as my own machine, and on the top-tier plans of several major AI providers.

Related questions

Which AI coding tool is best?

The one that measurably improves delivery for your codebase and team. Tools change quickly, so choose by measured effect rather than reputation.