From Engineering Harness to Cross-Functional Workflow
DoorDash’s GenAI platform team shifted evaluation from a technical bottleneck to a team sport. By treating evals as a continuous loop rather than a one-off engineering task, they enabled non-technical staff—strategy, operations, and product managers—to define quality standards. The platform team provides the infrastructure (telemetry, datasets, and judge APIs), while domain experts own the actual quality bar and rubric definitions. This organizational design is intentionally flexible; which team owns a specific judge prompt varies, reflecting an evolving understanding of how to best manage AI quality.
The API-First, Workflow-First Strategy
To avoid becoming a bottleneck, the platform team stopped trying to build custom UIs for every internal use case. Instead, they focused on shipping stable, robust APIs. This allows product teams and operations staff to use coding agents to "vibe code" their own annotation UIs tailored to their specific needs (e.g., image review vs. menu grading). This workflow-first approach empowers operators to build tools that fit their existing processes, significantly reducing the friction of back-and-forth communication with the central engineering team.
The Continuous Quality Loop
DoorDash operates on a repeatable eight-step cycle to maintain model quality:
- Trace: Capture agent/LLM sessions.
- Sample: Select a manageable subset of traces for human review.
- Annotate: Use domain-specific expertise to label data.
- Review: Validate the annotations.
- Golden Set: Promote high-quality data to a "golden set" for benchmarking.
- Calibrate: Use the golden set to tune LLM-as-a-judge prompts.
- Monitor: Track performance over time.
- Repeat: Iterate based on new data.
Self-Serve Calibration
To remove engineering dependency, the team built a self-serve UI for judge prompt calibration. This interface allows non-engineers to run optimization loops (using libraries like DSPy) and compare original versus optimized prompts side-by-side. By visualizing the changes, product managers can build trust in the automated judges. This democratization of the evaluation process has led to a sharp reduction in per-annotation costs and increased the velocity at which teams can ship reliable AI features.