Happy Sunday and welcome to Investing in AI! Check out this week’s AI in NYC show, and our new newsletter filled with AI/NYC tips and stories. Also be sure to check out Neurometric if you are at a stage in your AI lifecycle to care about token optimization.
Two engineers gave an AI agent the same one-sentence instruction: build a retro video game maker. The first run took twenty minutes and cost nine dollars. The second took six hours and cost two hundred — roughly twenty times more. If you stopped reading there, you’d assume the cheap run was the smart business call. But when Anthropic’s team actually opened the two applications, the nine-dollar version had a broken core: you could build a level, but you couldn’t play it. The two-hundred-dollar version worked. Same model. Same prompt. The only difference was the system wrapped around the model — what’s come to be called the harness.
That gap is not an edge case. In one reported project, adding a harness took an agent from a 20% success rate in a bare repository to nearly 100% — again, without touching the model. The lesson for anyone running a technology organization is blunt: the capability you can actually ship is not a property of the model you license. It’s a property of the model plus the system you build around it. And that system is the part you control.
This is why “harness engineering” has moved, in the span of a few months, from a niche term to a leadership-level concern. Here’s what it is, what it can and can’t do, and the tradeoffs your product and engineering leaders are now making whether or not they’re naming them.
What is a harness?
Strip away the jargon and a harness is everything around the model that governs how it sees the work, acts on it, gets feedback, and proves it’s finished. The shorthand making the rounds — “Agent = Model + Harness” — captures it well. In practice a harness is a handful of cooperating parts: instructions that tell the agent what to do and in what order, scope control that keeps it working on one thing at a time, memory that persists across sessions, verification that decides whether work is genuinely done, and a lifecycle that starts clean and hands off clean.
What a harness is not is a better prompt. A prompt can’t remember what happened in the last session, can’t run your test suite, and can’t stop an agent from declaring victory on a feature that doesn’t work. Those are systems problems, and they need systems, not cleverer wording.
What harnesses can and can’t do
Start with what they can do, because it’s the part that gets underestimated. A harness closes the distance between an impressive demo and a product you can ship. Benchmark scores flatter agents — the strongest ones land around 50 to 60% on curated tasks with clean descriptions, and lower in messy real-world repositories. The harness is where that gap gets closed, and crucially, you can build it in-house. You don’t have to wait for a vendor’s next model to get more reliable.
Now the uncomfortable half. Harnesses can’t yet guarantee that software does what you meant it to do. The current state of practice leans heavily on AI-generated tests to check functional behavior, and those aren’t trustworthy enough to remove human review. The dominant failure mode is well documented: agents tend to declare a job finished prematurely, and they’ll confidently praise their own mediocre work when asked to grade it. The structural fix that’s emerged — used by Anthropic and others — is to separate the agent that does the work from the agent that judges it; it turns out to be far easier to tune a skeptical, standalone evaluator than to make a generator honest about its own output. But even that doesn’t eliminate oversight. A good harness redirects expert attention to where it matters most; it doesn’t delete the need for it.
The strategic tradeoffs in harness engineering
Like most areas of tech, harness engineering has tradeoffs, because none of these decisions has a free answer. Here are the key tradeoffs to think through.
Cost versus quality is nonlinear. Recall the nine-dollar app. The real choice your teams face isn’t cheap versus expensive — it’s cheap-and-broken versus expensive-and-shippable. A twenty-times cost multiplier is worth it for the workloads where correctness is the product, and indefensible for the ones where it isn’t. Someone has to decide, per class of work, which tier is justified. That’s a product decision, not an engineering footnote.
Deterministic checks versus AI judgment. Harness controls come in two flavors with very different economics. Deterministic ones — tests, linters, type-checkers — are cheap, fast, and can run on every single change. The inferential ones — AI code review, an LLM acting as judge — add real semantic judgment but are slower, pricier, and non-deterministic. The practical consequence is a rationing problem: the cheap checks run constantly, while the expensive AI-driven ones get reserved for where they earn their keep. It’s worth knowing that the harness’s orchestration and its deterministic checks run on ordinary CPUs; only the model calls it makes reach for a GPU. So “how much inferential checking can we afford?” is, underneath, a question about GPU spend.
Generality versus governability. Not every system is equally amenable to being harnessed. A strongly typed language hands you a type-checker as a free sensor. Clear module boundaries let you enforce architectural rules. Opinionated frameworks quietly prevent whole categories of mistakes. Commit to a narrow, well-defined system shape and you can build a genuinely comprehensive harness around it; leave everything open-ended and you can’t. The strategic implication is new and worth saying out loud: your technology-stack and architecture choices are now also choices about how governable your agents will be.
Greenfield versus legacy. New projects can bake governability in from day one. Legacy systems carry the cruel irony that the harness is hardest to build exactly where it’s needed most — in the tangled, undocumented codebases that would benefit from it the most.
The decision the conventional wisdom gets wrong
There’s a tempting piece of conventional wisdom forming around harnesses: that they’re a depreciating asset. The reasoning is real. When Anthropic moved from one model generation to the next, they were able to delete scaffolding the older model had required — because every part of a harness encodes an assumption about what the model can’t do on its own, and those assumptions go stale as models get better. By that logic, harness work you do today is partly wasted the moment a stronger model ships.
But that logic quietly assumes you’ll always ride the upgrade curve. Many teams won’t — and shouldn’t. There are sound reasons to stay on an older, smaller, or cheaper model: cost per token, latency, throughput, data privacy, on-premise constraints, or simply the value of behavior you’ve already validated and don’t want to re-test from scratch. The moment you decide to hold the model fixed, the depreciation argument inverts. The scaffolding that compensates for a smaller model’s weaker planning and shakier self-verification doesn’t go stale — it stays fully load-bearing, precisely because you’ve chosen not to buy your way out of those weaknesses.
So the sharper framing is this: a harness depreciates only to the extent that you keep buying your way up the model curve. Choose to stay small on purpose — for cost, for speed, for control — and the harness becomes one of the most durable assets you own. And even for teams that do chase the frontier, the useful insight is that the space of valuable harness designs doesn’t shrink as models improve. It moves outward, toward more ambitious tasks the raw model still can’t do alone. The question was never whether to invest in a harness. It’s which frontier you’re investing against.
Three practical takeaways
Now that you understand harness engineering, what should you think about if you are a tech executive that needs to make some decisions in this area?
First, treat your harness as a portfolio asset with a depreciation schedule — but set that schedule according to your own model strategy, not your vendor’s release calendar. If you’re deliberately staying on a smaller model, your harness isn’t depreciating at all.
Second, add a governability lens to stack and architecture decisions. The question “how easily can we harness agents working in this system?” now belongs alongside cost, performance, and team familiarity.
Third, start with verification and observability. They’re the cheapest pieces to build and they return the most, because they attack the failure mode — agents that confidently ship broken work — that quietly costs you the most.
The model sets your ceiling. The harness determines how much of that ceiling you actually ship. And on a model you’ve chosen to hold fixed, the harness isn’t part of the game — it’s the whole game.
Thanks for reading.