============================================================ nat.io // BLOG POST ============================================================ TITLE: The Model Is Not the Unit of AI Performance DATE: July 21, 2026 AUTHOR: Nat Currier TAGS: AI, AI Engineering, Evaluation, Systems Thinking ------------------------------------------------------------ The procurement spreadsheet says Model A wins. It has a 2.4-point lead on the benchmark the engineering team trusts. The price difference is tolerable. The vendor presentation turns the lead into a simple claim: better model, better software. By the time the meeting starts, the ranking already feels like a decision. But the production system will not run a leaderboard. It will run company repositories with private dependencies, uneven test coverage, intermittent services, security rules, time limits, and developers who care whether a change is maintainable. It may give the model different tools, fewer retries, more context, less memory, or a grader that values the wrong thing. The benchmark score can be accurate and the procurement conclusion can still be wrong. That is because the model is not the unit of AI performance. Observed performance belongs to a composed system: the model, task contract, harness, environment, budget, and grader. Change one of them and you can change the result without changing the model. A leaderboard is therefore not a deployment verdict. It is evidence about one system under one set of conditions. This does not make benchmarks useless. It gives them a more precise job. Public evaluations generate hypotheses about what a model may do well. Deployment-shaped acceptance evidence decides whether the system you can actually operate is fit for the work you actually have. [ A score is the output, not the explanation ] ------------------------------------------------------------ A benchmark compresses many experimental choices into one visible number. That compression is useful. It lets people compare results quickly, watch progress, and identify candidates worth deeper investigation. The danger begins when the compressed number is treated as though it were a property stored inside the model. A coding score is not produced by weights alone. The system must decide what task to present, which instructions and repository state the agent can see, which tools it can call, how long it can run, what happens when a command fails, and which tests count as success. Even the same nominal model can behave differently with another reasoning setting, context policy, or retry loop. A more honest representation is: `observed performance = f(model, task contract, harness, environment, budget, grader)` This is an accountability map, not a claim that each term has a stable mathematical weight. It tells us where a result can change and what an evaluation report must make inspectable. | Component | Decision hidden inside the score | Example | | --- | --- | --- | | Model | Which model, version, reasoning mode, and sampling configuration ran? | A fixed model with a high reasoning setting | | Task contract | What behavior was requested, what context was supplied, and what counted as in scope? | Implement a feature without changing a public interface | | Harness | How were tools, state, retries, context, and error recovery managed? | Preserve terminal state and retry a failed dependency install | | Environment | What hardware, repository, network, services, and permissions existed? | Four CPUs, 16 GB RAM, no internet, private package mirror | | Budget | How much time, compute, money, context, and how many attempts were allowed? | Twenty minutes, two attempts, fixed token ceiling | | Grader | Which observable conditions converted the run into pass or fail? | New tests pass, old tests remain green, no security regression | Calling the final number a “model score” drops five parts of the experiment from the noun. That shortcut was less damaging when an evaluation was a prompt, a response, and a narrow answer key. It becomes dangerous when an agent modifies state across a long trajectory. [ The task can be broken while the score remains precise ] ---------------------------------------------------------------- In July 2026, OpenAI published [an audit of SWE-Bench Pro](https://openai.com/index/separating-signal-from-noise-coding-evaluations/), a benchmark designed to test longer and more realistic software engineering work. The 731-task public split had shown frontier-model pass rates rising from 23.3 percent to 80.3 percent in eight months. That looks like an unusually clear capability curve. The audit found a less comfortable measurement story. An automated analysis pipeline flagged 200 tasks, or 27.4 percent, as broken. A separate human annotation campaign identified 249, or 34.1 percent. OpenAI estimated that roughly 30 percent of the tasks were broken and retracted its earlier recommendation to adopt the benchmark. The failure categories matter more than the headline percentage: - Some tests enforced implementation details the prompt never required, causing functionally valid solutions to fail. - Some prompts omitted requirements that hidden tests enforced and that could not reasonably be inferred. - Some tests covered too little of the requested feature, allowing incomplete implementations to pass. - Some prompts pointed toward behavior that contradicted the grader. In each case, the score still had numerical precision. The measurement contract did not have conceptual validity. This produces two symmetrical errors. An overly strict grader can make a capable system look weak by rejecting valid work. A low-coverage grader can make an incomplete system look strong by rewarding a patch that satisfies the visible test without fulfilling the request. Better model capability does not repair either error. In fact, as models improve, benchmark defects can become a larger share of the remaining failures and passes. The task contract and grader form a pair. The contract says what work is required. The grader decides what evidence proves that work occurred. If they disagree, the benchmark is measuring adaptation to the disagreement. The same defect appears inside companies whenever an acceptance suite rewards what is easy to count rather than what the task requires. A benchmark-specific measurement problem becomes a product decision problem as soon as the score is used to select a system. The number remains real. Its meaning changes. [ The same model can gain six points from the machine around it ] ----------------------------------------------------------------------- The task and grader are not the only participants. Anthropic's February 2026 study, [“Quantifying infrastructure noise in agentic coding evals”](https://www.anthropic.com/engineering/infrastructure-noise), held the model, orchestration setup, and Terminal-Bench 2.0 task set constant while changing resource configurations. The gap between the most constrained and uncapped setups was six percentage points. Under strict enforcement, infrastructure errors reached 5.8 percent. With uncapped resources, they fell to 0.5 percent. More headroom first prevented spurious container failures, then began enabling strategies that could not run under tighter limits. An agent could install large dependencies, spawn expensive subprocesses, or execute memory-intensive test suites. Nothing about the model weights changed. The result does not imply that uncapped infrastructure is the fair setting. Tight limits may be exactly right when a production workflow has hard cost or latency constraints. Under those conditions, an agent that finds a lean solution should outperform one that requires a large toolchain. Generous limits answer a different question: what can this system accomplish when resource efficiency is not the binding constraint? Both questions are legitimate. They cannot share one interpretation. Anthropic recommends skepticism toward leaderboard differences below three percentage points when the resource methodology is not documented and matched. This matters because public rankings often separate leading systems by margins smaller than the variation introduced by infrastructure. The practical lesson is not merely “control the servers.” It is that an agentic evaluation is an end-to-end system test by construction. CPU, RAM, timeouts, network access, cluster health, tool availability, and dependency state can all participate in the observed behavior. When the environment is omitted from the report, readers cannot tell whether a small lead came from a stronger model, a more favorable operating condition, or both. [ The harness is part of the capability you observed ] ------------------------------------------------------------ This harness is the control system between the model and the task. It assembles context, exposes tools, maintains state, parses actions, handles failures, decides whether to retry, and determines when the run ends. Those choices can look like evaluation plumbing. They can also determine whether a capability appears at all. OpenAI's [playbook for trustworthy third-party evaluations](https://openai.com/index/trustworthy-third-party-evaluations-foundations/) makes this explicit. A setup that preserves state and retries failed actions may enable a model to complete a multi-step task that the same model does not finish in a simpler setup. The playbook argues that reports should state the claim being tested and provide evidence that the evaluation result is valid. It also recommends reporting the tested system, tool access, orchestration setup, safeguards, interaction and inference budgets, elicitation methods, and checks for hazards such as contamination, reward hacking, broken problems, refusals, and sandbagging. This creates an important distinction between capability and product fit. A strong elicitation setup is appropriate when the question is, “Can this model produce the capability under credible favorable conditions?” The evaluator may invest in careful prompting, state management, retries, and tools to find the system's ceiling. A deployment-shaped setup is appropriate when the question is, “Will this configuration perform inside our operating constraints?” It should reproduce the tools, permissions, budgets, error paths, and context policies the product can sustain. Neither setup is more honest in the abstract. Honesty comes from matching the setup to the claim. A team gets into trouble when it buys against a capability-ceiling result and budgets for a minimal production control layer. The benchmark may have demonstrated something real, but the purchased operating system cannot reproduce the conditions under which it appeared. [ The grader defines what victory means ] ------------------------------------------------------------ Evaluation discussions often spend more time on prompts than graders. This is backwards. The grader is where product values become operational. For a coding task, a test suite can verify functional behavior cheaply and reproducibly. It may not notice an unnecessary dependency, an authorization bypass, a duplicated abstraction, or a patch that is technically correct but impossible for the team to maintain. An exact-match grader is even narrower. A model-based grader can recognize more valid variations, but it introduces its own judgment errors and calibration work. Human reviewers bring context and responsibility, but they are slower and can disagree. Anthropic's [guidance on agent evaluations](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) recommends combining code-based, model-based, and human graders according to the work. Code checks are fast and reproducible but brittle. Model graders can evaluate qualities that resist exact rules but need validation against human judgment. Humans remain important for calibration and for consequences the automated layers cannot faithfully represent. The right grader is usually a stack: 1. Deterministic checks establish hard facts such as schema validity, test results, prohibited side effects, and required artifacts. 2. Model-based review assesses qualities such as instruction following, evidence coverage, or the coherence of a multi-step result. 3. Human review calibrates the other graders and decides cases where context, consequence, or legitimate disagreement matters. No layer converts judgment into certainty. The stack makes different failure modes visible. This is also why “pass rate” is incomplete without an invalid-run policy. If a service outage, broken sandbox, or missing dependency prevents the task from starting, does the run count as a model failure, get excluded, or trigger a retry? Each choice changes the denominator. A trustworthy report states the rule before presenting the result. [ A leaderboard is a hypothesis generator ] ------------------------------------------------------------ Public benchmarks still do valuable work. They create shared tasks, make some comparisons reproducible, expose capability trends, and prevent every team from starting with anecdotes. A large, carefully maintained benchmark can reveal patterns that a small internal suite cannot. For teams with limited evaluation capacity, leaderboards are a rational way to reduce a large model market to a manageable shortlist. The boundary is the claim. A benchmark supports a claim about performance on its task distribution under its reported conditions. It does not automatically support a claim about performance on your repositories, customers, policies, hardware, or cost ceiling. A two-point lead can justify investigation. It does not justify skipping acceptance testing. The useful posture is neither leaderboard worship nor benchmark cynicism. Treat the ranking as a prior belief. Then update that belief with evidence shaped like the deployment decision. This reframes model selection. The question is not “Which model is best?” It is “Which composed system provides the best acceptable operating point for this workload?” That operating point can include quality, latency, cost, reliability, security, data boundaries, observability, and the amount of human correction required. A model that wins an unconstrained capability test may lose when every task has a two-minute service objective. A smaller model may be the better default for routine cases while a frontier model handles difficult escalations. One model may excel only when paired with an orchestration layer your team cannot safely maintain. The decision belongs to the system you can operate. [ Build an evaluation claim card before you run the eval ] ---------------------------------------------------------------- The smallest useful artifact is an evaluation claim card. It prevents a score from outrunning the experiment that produced it. Here is a concrete version for a team selecting a coding agent for maintenance work: | Field | Declared condition | | --- | --- | | Decision | Select the default agent for bounded changes in three internal TypeScript services | | Claim | The tested system can complete representative maintenance tasks without regressions under the production budget | | Task distribution | 60 private recent bug fixes and small features, sampled by repository and change type | | Success contract | Requested behavior works, existing behavior remains green, no prohibited dependency or permission change, patch is reviewable | | Model configuration | Exact model snapshot and reasoning setting recorded | | Control layer | Repository search, patch editing, terminal, test runner, persistent state, one automatic recovery attempt | | Environment | Production-like container, fixed CPU and RAM, private registry access, no public internet | | Budget | 20 wall-clock minutes, two attempts, fixed inference-cost ceiling | | Graders | Existing and task-specific tests, lint and type checks, security policy checks, blinded maintainer review | | Invalid-run rule | Infrastructure failures reported separately and rerun once; never silently counted as task failure or success | | Acceptance rule | Quality floor must pass, then choose the lowest total cost among systems with acceptable latency and correction burden | This card does more than improve reporting. It exposes unresolved product questions, from workload scope and reviewability to network access, recovery, and grader disagreement. If the team cannot answer those questions, it is not ready to interpret a decimal score. [ Compare systems with deployment-shaped acceptance evidence ] -------------------------------------------------------------------- Once the claim is explicit, model selection becomes a compact engineering process. > Start with workload evidence Sample tasks from the work the system will actually perform. Preserve the distribution of ordinary cases, difficult cases, and consequential edge cases. Do not build a suite entirely from incidents, but do not remove the messy cases that distinguish production from a demo. Keep a holdout set for final comparison. If prompts, agent-loop behavior, or grader thresholds are repeatedly tuned against every task, the internal suite becomes another benchmark the team has learned to game. > Freeze the run manifest Record the model snapshot, reasoning setting, system instructions, tools, harness version, environment image, resource limits, retry policy, task ordering, and grader versions. A result without a manifest cannot explain a later regression. Equivalent conditions matter when comparing models. Deployment conditions matter when making a product claim. Ideally, the same manifest gives you both. > Grade the outcome, not the performance theater Prefer observable end state over stylistic traces. For coding, inspect the repository and behavior after the run. For research, verify claims against the retrieved evidence. For an operational agent, inspect the external state change and whether it respected authority boundaries. Transcripts are still useful. They explain how the system reached the outcome and help diagnose grader disagreement. They should not substitute for checking whether the work is correct. > Separate failure classes Report at least task failure, policy failure, grader uncertainty, and infrastructure-invalid runs separately. Collapsing them into “did not pass” hides which part of the composed system needs work. This is where the unit-of-performance view pays off. A task failure may justify another model. Repeated tool parsing failures may justify another control layer. Container crashes may justify environment repair. Reviewer disagreement may expose an incomplete task contract. A failing score does not name its own cause. > Choose an operating point, not a champion Require quality and safety floors first. Among systems that pass, compare cost, latency, variance, human correction, and operational complexity. Run enough repeats to expose instability where the consequence warrants it. The output may be a portfolio rather than one winner: a fast default, a high-capability escalation path, and a deterministic fallback for tasks that should never have been agentic. That is a stronger decision than forcing every workload through whichever model occupies the first row of a public table. [ Name the system you tested ] ------------------------------------------------------------ The next time a leaderboard appears in a model-selection meeting, use it. Let it narrow the field. Let it reveal a surprising candidate. Let it challenge an internal assumption. Then ask what produced the number. What task contract was the system given? Which orchestration setup elicited the behavior? What environment supported it? What budget funded it? Which grader defined success? How closely do those conditions resemble the ones your product can sustain? If those answers are missing, the ranking is still useful. It is a hypothesis waiting for an experiment. The model still matters. Naming the rest of the system makes its capability legible under the conditions that elicited it. Run that experiment against your work. Preserve the manifest. Inspect the failures. Choose the operating point that survives your constraints. Model selection becomes much less theatrical when the unit of performance is named correctly. It also becomes much more valuable.