Do Harnesses Matter? What seven Codex releases reveal about agent efficiency

We evaluate coding agents as one system, i.e., a model and its harness, and report a single performance number. This is fine if you want to pick the best agent. It's less useful if you want to understand what each component actually contributes. Every model release informs the capability of the model and provides directional insight into its capabilities. What about the harness? With recursive loops and auto harness designs, one aspect of harness engineering was unclear - what have the manual iteration of the harnesses achieved so far?

Broadly speaking, an agent system looks like:

Harness(model, input) -> (trajectory, artifacts)

The model is fixed. The input is a repo and an issue. The harness is the thing doing context construction, tool orchestration, retry logic, and context management. The trajectory tells you how it got there — every tool call, every compaction step, every token.

Coding benchmarks evaluate the artifacts produced and evaluate them (e.g., test pass rate, patch quality).

To answer what manual iteration of a coding harness achieves, in our experiment we fix the model and evaluate a single harness that has been iterated manually over time.

TL;DR - All harnesses produce nearly the same pass rate (accuracy). But the token efficiency and consequently the cost, computational effort is the key improvement over time.

The setup

We make use of randomly sampled 100 task slice from SWE-bench Multilingual and run each task across seven stable Codex releases from January to July 2026. That's 700 version-task evaluations.

The model was pinned to gpt-5.6-sol with reasoning effort set to high. Each run was sandboxed in a docker and run with the same setup of repo snapshots, issue text, and prompt. Note that a change of codex version encompasses the full bundle of the harness prompts, tool descriptions, execution policy, retry logic, and context management. The code setup and evaluation follows the methodology described in the SWE-bench documentation.

For each run, we tracked two things:

• What did it cost? input tokens, output tokens, and estimated API cost across the full agent trajectory.
• Did it work? Binary pass/fail based on the artifact produced.

Now to the results.

Pass rate? Basically flat.

Resolution rate across seven Codex CLI releases.

Aggregate scores often hide interesting rearrangements in which tasks get solved. So digging deeper at tasks directly showed 76 out of 100 tasks have identical outcomes under every release. Of the remaining 24, the newest release gains five tasks and loses five relative to the oldest. Note that these are single run observations, so the flips could just be an effect of the randomness in model generations.

Pass and fail outcomes for the 24 tasks that change across Codex releases.

The flatness of the pass rate says more about the model than the harness: if the model is strong relative to the interface changes being made, the pass rate won't reveal much about the harness itself. A coding harness needs to let the model inspect files, search the repo, edit code, run tests, and return a patch. There are many ways to implement these operations. If the model can compensate for differences among them, two harnesses can produce very different trajectories while reaching patches of similar quality.

This doesn't mean the harness is irrelevant. It means pass@1 isn't resolving its effect here. Similar observations have shown up in fixed-model comparisons like Does the Harness Matter?: model choice dominates the score, harness choice shows up in tokens, cost, and time.

The trajectory tells a different story

Here's where it gets interesting.

Token counts over the complete agent run shows meaningful differences across releases. Mean input tokens (cached + uncached) usage goes from 0.55M tokens per task in v0.80.0 to 1.12M in v0.98.0, stays near that level through v0.120.0, then falls below 0.9M in the last two releases. Output tokens follows the same arc: 5.9K → 8.5K → back to ~6–7K.

Left: Total input, cached input, and output tokens, Right: Estimated API cost per task across seven Codex releases.

Output tokens are a useful proxy for the serial work the model does: reasoning, commands, edits, responses. Later harnesses reach the same pass rate with shorter generated trajectories than the bloated middle releases.

Comparing the middle phase (v0.98.0–v0.120.0) against the later releases (v0.130.0–v0.144.6) task by task, the median changes:

Not a handful of expensive repos driving this — the improvement is broad. The harness went through a phase of bloat followed by recovery. Version 0.80.0 is still the leanest in the experiment.

So what?

This is one harness on one benchmark. But coding is coding. The tasks here — inspect a repo, understand an issue, edit code, run tests — are the same operations any coding agent performs in production. The finding generalizes at least directionally: manual harness iteration made the agent cheaper, not more capable.

That distinction matters because benchmarks don't see it. Two harnesses with the same pass rate look identical on a leaderboard. But if one burns twice the tokens getting there, it costs twice as much, takes longer to respond, and creates more surface area for tool errors and context loss along the way.

The benchmark sees these systems as equivalent. A product serving real users does not.

Seven months of Codex releases produced a measurable correctness trend on this task panel. The harness evolved less in what the model could finish than in how much work it asked the model to do. If this is true, the next question is what automated harness improvements would do and if they lead to bloat, which I suspect happened with codex after its initial release and if that is the wrong thing for the system to recurse on.

---

This post is based on a controlled experiment run across seven Codex CLI releases. Code is available at github.com/shekkizh/.