The Gap Between Demos and Real-World Adoption
Technology diffusion—like the transition from steam to electricity—takes generations because it requires ripping out old infrastructure and retraining entire workforces. Current AI development suffers from a 'demo bias' where agents perform well in controlled environments but fail to integrate into the messy, serial nature of real-world services. To bridge this, builders must stop acting as external vendors and start acting as operator-owners who are responsible for the actual outcomes (e.g., whether a roof is repaired or books are closed).
From Co-Pilot to Co-Worker: The Autonomy Ladder
Moving from a simple RAG-based co-pilot to an autonomous co-worker requires earning trust through a structured ladder of autonomy:
- Co-pilot: Quick information retrieval.
- Synchronous Agent: Real-time interaction with tools.
- Asynchronous Agent: Background execution triggered by events rather than user queries.
- Long-running Agent: Multi-day or multi-week task management.
Engineers are uniquely comfortable with asynchronous, parallelized work (launching 10 jobs and accepting non-linear completion). However, most service industries operate serially. The challenge is representing complex knowledge work as code and building the infrastructure to parallelize tasks that have traditionally been handled one-by-one.
The Flywheel of Real-World Evals
Most valuable tasks—like scoping a building or coordinating vendors—are not documented on the internet; they live in the heads of senior operators or legacy software. By embedding agents directly into these businesses, builders can capture 'rich traces' of data, including tool calls, errors, and paper cuts. This creates a ground-truth dataset that allows for:
- Automated Evals: Scoring agents based on real-world outcomes rather than synthetic benchmarks.
- Internal Post-Training: Fine-tuning models on proprietary data that is out-of-distribution for frontier labs.
- Regression Testing: Turning every failure into a permanent test case to ensure the agent 'hill climbs' toward better performance over time.
Co-Design and the 'Touch Grass' Strategy
Adoption is the primary bottleneck. A superior AI tool will fail if it doesn't fit into the existing habits of a 100-year-old firm. The solution is 'extreme software-service co-design,' which cannot be achieved over Zoom. It requires:
- Physical Presence: Showing up at trade conferences, running stands, and observing workflows in person.
- Native Integration: Building tools directly into the software operators already use (Excel, ERPs, Outlook).
- Unified Loops: Treating continual learning (research/engineering) and enablement (growth/adoption) as a single, inseparable loop. Usage drives data, which drives better models, which drives further usage.