ai-agents

You're Arguing About the Wrong Layer

Researchers ran three models through three harnesses on 100 coding tasks. The harness moved performance almost eight times more than the model did, and the model rankings flipped six times out of nine. Your model-choice argument is measuring the scaffolding.

Johan Kölhi

Johan Kölhi

— 6 min read
Share
You're Arguing About the Wrong Layer

Somebody on your team has an opinion about which model is best for coding. They have benchmarks. They have receipts.

They also have a problem. A team of researchers ran three frontier models through three harness configurations on 100 SWE-bench tasks. Across nine model-pair comparisons, the ranking flipped six times. Which model "won" depended on the scaffolding wrapped around it.

Same models. Same tasks. Different answer.

That result comes from an arXiv paper with a blunt title: Stop Comparing LLM Agents Without Disclosing the Harness. The headline number is the variance ratio. Harness-induced performance variance was roughly 7.8 times larger than model-induced variance. The thing everyone argues about mattered least.

What a harness actually is

The formula shows up in nearly every serious source on this: Agent = Model + Harness.

Databricks defines the harness as the software infrastructure wrapped around the model that lets it act on tasks rather than just respond to prompts. Eight parts: system prompts, tools and tool execution, sandboxes, filesystem and storage, memory and context management, feedback and self-verification loops, guardrails and human checkpoints, and observability.

Faros.ai compresses that to five layers and puts it on a maturity curve. Prompt engineering, then context engineering, then harness engineering. We are in the third phase now.

Everything in that list is code you write. None of it comes from your model vendor.

The numbers hold up

The variance ratio is not one odd study. The same paper compiles field evidence pointing the same way.

Hold Claude Opus 4.5 constant on SWE-bench Pro, change only the harness, and performance swings 9.5 points, from 45.9% to 55.4%. Independent monitoring of SWE-bench Verified found up to 15 points of scaffold-only variation. On Terminal-Bench, harness evolution alone added 7.3 points.

Then the comparison that should end the argument. LangChain moved Terminal-Bench 2.0 from 52.8% to 66.5% on identical model weights. That is 13.7 points from scaffolding. The Claude Opus 4.5 to 4.7 generation upgrade was worth about 6.8 points. Better plumbing beat a full model generation, twice over. Teams have climbed from outside the top 30 into the top 5 this way.

The structural explanation is the interesting part. Agents are closed-loop systems. The harness implements the controller that updates state. The model is an open-loop policy inside it. Swapping the model changes the policy. Rewriting the harness changes the control system. Control systems dominate.

The version that survives a budget meeting

Engineers find this interesting. Finance finds it urgent.

A second controlled study, this one across six foundation models and 22 locked evaluation tasks, changed only the execution layer and held everything else fixed. Tokens per task dropped 38%. Cost per task dropped 41%. Median latency dropped 44%. Task completion scores went slightly up, not down.

Quality per dollar improved 82%. Every one of the six models improved, in a range of 33% to 61%. The authors call it a property of the orchestration layer, not of any model.

Now put that next to the decision most teams actually make. Switching from the most expensive model on the menu to the cheapest saves about 36%. Redesigning the harness saves 33% to 61%, and it saves it with whichever model you were already using.

You have been shopping for a cheaper model when the money was in the plumbing.

Two mechanisms drive most of it. First, prompt structure. Writer restructured prompts so the prefix stayed stable across calls and got 99.9% of tokens served as cache reads, billing at roughly a tenth of list price. Second, routing. One legal document pipeline cut per-document cost from $1.40 to $0.34 by sending four of seven steps to mid-tier models and two to small models, keeping only the final step on a frontier model. Routing is a harness capability, not a model feature.

Your agent fails for reasons no model upgrade fixes

Anthropic published a taxonomy of how long-running agents break. Read it and notice how little of it is about intelligence.

Victory declaration: the agent marks the task done without verifying anything. One-shotting overreach: it tries to build the whole thing at once and leaves half-finished code. Undocumented progress: it leaves the environment broken for the next session. Context anxiety: the model starts rushing as the context window fills.

None of those get fixed by a smarter model. A smarter model declares victory more convincingly.

Anthropic's own fixes are architectural. Split the work between an initializer agent and a coding agent. Write a JSON feature list with 200 or more specific requirements. Ship an init.sh that rebuilds the environment. Keep a progress log. Commit to git as checkpoints. Require browser automation to verify a feature before anything gets marked complete.

Their framing line is the one worth keeping: each new engineer arrives with no memory of what happened on the previous shift.

That reframes the whole problem. A context window is a shift handover. We have known how to run reliable shift work for a century, and none of it involves hiring smarter people. It involves checklists, handover logs, verification steps, and a written definition of done. Faros puts the operating principle plainly: anytime you find an agent makes a mistake, engineer a solution so the agent never makes that mistake again.

Where model choice still counts

The absolutist version of this argument is wrong, and your engineers will spot it.

The model sets the ceiling. The harness determines how close you get to it. Some tasks genuinely need reasoning that only stronger models have. Push a weak model through harness evolution and you hit a scaffolding ceiling, where the tools and context and control flow are all in place and the frozen model simply cannot use them.

Model and harness are also fusing. Models get post-trained against specific harnesses, which bakes tool preferences into the weights. Either model can use either tool, but handing it the unfamiliar one costs extra reasoning tokens and produces more mistakes. Swapping models is not free either.

The practical rule: optimize the harness first, upgrade the model when you hit a real ceiling.

Two honest caveats. Almost every vendor writing about this sells harness tooling, which makes "the harness matters more" a conveniently self-serving conclusion. The two arXiv papers are the load-bearing evidence. And nearly all the numbers come from benchmarks, not from messy internal workflows. Whether a 7.8x ratio survives contact with your legacy ERP integration is genuinely untested.

What to do on Monday

Stop asking which model is best. Start asking what your harness does when the model is wrong.

Write down your harness configuration across seven layers: execution, tools, context, scheduling, observability, verification, governance. The paper calls this a Harness Card. Most teams cannot fill one in, which is the finding.

Add one verification loop. Pick your most common agent failure and engineer it out. Measure recovery rate and context retention, not just pass rate.

And the next time someone shows you a leaderboard for agentic work, ask which harness produced the number. If they cannot tell you, the number is not about the model.

Sources

Read more