Case study · AI in production
An AI engineering team that has to show its evidence.
Our own orchestrator runs a small team of AI agents on a real codebase: they take specs, open pull requests, wait for real CI and merge real code, unattended. The interesting part is everything that stops them merging when they shouldn't.
01 The situation
Getting an AI model to write code is the easy part now. Leaving it alone with a repository overnight is not.
The harness splits the work into roles. A Planner breaks a spec into stories, an Engineer builds each one in its own branch, and a Closer decides whether it ships. The owner sets the order of the backlog and steps in on strategy, not on every pull request.
02 The risk we set out to remove
An unsupervised agent fails in quiet ways. It says the tests pass when they didn't run. It retries a broken story until the budget is gone. It crashes halfway and loses the work, or merges something that should have waited for another change.
The goal was never "an AI wrote some code". It was a pipeline that keeps running correctly when a story fails.
03 How we approached it
-
i
Ship on evidence, not on say-so
The Closer certifies work against evidence rather than defaulting to ship. Verification runs on the repository's own CI and required checks, not on the agent's word. A rejected story comes back with exactly what evidence was missing.
-
ii
Make every failure recoverable
A crashed run's unfinished work is checkpointed and resumed on the next attempt. A bad merge can be reverted durably, even if the process dies mid-revert. A story that keeps failing is quarantined instead of retried forever, and the failure count survives a restart.
-
iii
Escalate deliberately, and in one direction
A story that fails on a cheaper model moves to a stronger one and never drops back. A story that depends on another waits until that one has merged, without holding up unrelated work.
-
iv
Keep a human on what the agents learn
Agents propose lessons at the end of each run. Only a person can promote one into the knowledge the next run starts from, so one confident mistake can't become house style.
04 What it achieved
-
A merge with zero human steps
In July 2026 the harness took a story on a separate production codebase from spec to pull request, through green CI, to a merge — without a person touching it.
-
Building itself, unattended
Pointed at its own repository — tens of thousands of lines and a real CI job — its three most recent batches each merged at least two-thirds of their stories unattended.
-
Recovery proven against the real thing
The revert path passed every case of a live contract test against real GitHub, including a crash at each stage of the revert.
05 What went wrong on the way
The early self-build runs merged nothing. The cause was the measuring instrument, not the agents: the Closer was comparing each story against a stale branch, so other merged work showed up in its diff and correct changes were bounced as out of scope.
One recovery feature was shipped and then deliberately removed, because it could overwrite live work and no safe version of it existed. We would rather publish that than pretend every idea held up.
06 Contact
Want AI agents doing real work in your codebase, with guardrails you can explain to an auditor? Get in touch.
connect@happy.software