From Token Leaderboards to Smoke Detectors

Many organizations mistakenly treat AI token usage as a performance metric, creating leaderboards that incentivize wasteful consumption. Mingsheng Hong argues this is a critical error. Instead, dashboards should function as smoke detectors: high usage might be normal for certain teams, but unexpectedly low usage or sudden spikes should trigger a conversation. Like "lines of code" in traditional software, token spend is a metric to track, not a goal to optimize. The objective is not austerity, but maximizing the Return on Investment (ROI) of every token spent.

Defining Trusted Throughput

To measure ROI, teams must look beyond cost and track value. Ironclad uses the concept of trusted throughput—work that is validated through three distinct layers:

  1. Objective Metrics: Automated test coverage, security scans, and canary deployment success.
  2. Subjective Human Judgment: Code reviews focusing on architecture, maintainability, and design quality.
  3. Customer Impact: Real-world performance, lack of production rollbacks, and positive user feedback.

To quantify this, the team evolved their metrics from tracking "open pull requests" (which AI makes abundant) to "merged pull requests" weighted by a complexity score. This complexity score is generated by feeding PRs into an LLM with a prompt that assigns a t-shirt size, ensuring that a 10-line concurrency fix is valued appropriately against 1,000 lines of boilerplate.

Addressing the New Bottlenecks

AI-driven code generation shifts the bottleneck from writing code to reviewing and merging it. When CI pipelines are slow or flaky, engineers are incentivized to submit massive, batched PRs to avoid repeated wait times, which degrades review quality and increases risk. To counter this, engineering leaders should:

  • Offload Review: Use AI as a first-pass filter for style and test coverage, allowing human reviewers to focus on deep architectural judgment.
  • Fix CI Infrastructure: Treat flaky tests and slow pipelines as critical blockers. Measure the "wait time" from PR readiness to merge as a key developer experience metric.
  • Control Agentic Loops: When using AI agents to auto-fix code or tests, implement hard caps on retry loops to prevent runaway token consumption.
  • Optimize Context: Encourage "muscle memory" for prompt caching (placing fixed system prompts at the top) and context pruning (summarizing long sessions) to improve efficiency and output quality.