An agent harness can fail because it works.
Suppose a model tends to lose the plot near the end of a long task. The team responds sensibly. It adds a planner, divides the work into sprints, resets context between phases, asks a second model to evaluate each result, and retries anything below a threshold.
Completion improves. The surrounding system becomes production infrastructure. Its behavior is documented, monitored, and copied into the next workflow.
Then the model changes.
The new model plans more coherently, retains context longer, and catches mistakes the evaluator was built to find. The old scaffolding does not disappear. It still decomposes the work, discards context, inserts latency, consumes tokens, and gives the evaluator another chance to reject a valid result.
The team sees a more capable model running inside a system designed around its predecessor's weaknesses. When the result disappoints, everyone tunes the prompt.
The real problem is architectural: the scaffolding preserved a limitation after the limitation moved.
An agent harness is not neutral plumbing. Every planner, retry rule, context transform, grader, and handoff encodes a claim about what the model can be trusted to do. Those claims age. Some expire quickly. The surrounding system therefore has a half-life: the period over which its assumptions remain useful enough to justify their cost and constraint.
The practical consequence is not that teams should remove orchestration whenever a better model appears. It is that orchestration code needs the same skepticism we apply to compatibility shims, feature flags, and performance workarounds. A component should survive because current evidence shows that it still improves the system, not because it once rescued a weaker model.
Scaffolding is an executable theory of weakness
The agent loop prepares context, selects tools, routes calls, records state, evaluates progress, handles failure, and decides whether to continue. That surrounding system is often what turns a promising model demonstration into reliable work.
But the word _harness_ hides two different kinds of architecture.
The first kind establishes durable system boundaries. Sandboxes limit what generated code can reach. Session logs preserve what happened. Credential vaults keep secrets outside the model's environment. Stable tool interfaces separate reasoning from execution. These controls remain useful even when models become more capable because they address security, recovery, and operability at the system level.
The second kind compensates for observed model behavior. A planner exists because the generator under-scopes. A context reset exists because long tasks decay. A second-pass grader exists because the first pass misses certain defects. A retry policy exists because one attempt is not dependable enough.
These components can be valuable. They are also empirical bets.
Anthropic describes a direct example in its Managed Agents architecture. Claude Sonnet 4.5 tended to wrap up work as it approached its context limit, so the orchestration layer added context resets. With Claude Opus 4.5, that behavior was no longer present. The resets had become dead weight.
In a separate account of long-running application development, a multi-stage control loop helped an earlier model plan and evaluate a complex build. When Opus 4.6 improved long-horizon planning, retrieval, code review, and debugging, parts of the same structure became less useful. The team did not guess which parts to delete. It removed components individually and examined the effect.
That is the right mental model. A workaround is a hypothesis made executable:
Under this task distribution, with this model and tool environment, component X improves outcome Y enough to justify cost Z.
Change the model, distribution, environment, outcome, or cost, and the hypothesis needs to be tested again.
Better models do not make every control loop smaller
It is tempting to turn this into a simple prediction: models improve, so scaffolding shrinks.
Reality is less tidy.
A stronger model may need less decomposition but receive access to more tools. It may require fewer retries while operating across longer tasks. It may make a planner unnecessary but create a greater need for durable state, permission boundaries, and trace inspection. Better reasoning can remove cognitive scaffolding while increasing the importance of operational controls.
The boundary also moves unevenly. A model may handle ordinary application work without a grader but still need independent checks for authentication, money movement, destructive infrastructure changes, or subtle visual defects. The same evaluator can be wasteful on one task class and essential on another.
This is why capability claims at the model level are insufficient. Scaffolding utility is conditional on the work.
OpenAI's account of building an agent-written product makes this visible from another direction. Its team invested in repository-local knowledge, executable plans, linters, observability, and feedback loops because anything inaccessible to the agent effectively did not exist. Those investments are not merely patches for a weak model. They make the environment legible and the result inspectable. A future model may read the repository more effectively, but it still cannot act on a decision trapped in a private conversation.
The useful distinction is not small orchestration layer versus large orchestration layer. It is durable control plane versus perishable compensation.
| Runtime element | Primary job | Likely durability | What should trigger review |
|---|---|---|---|
| Sandboxed execution | Bound blast radius | High | Threat model or access model changes |
| Durable session log | Preserve state and recovery evidence | High | State model or retention requirements change |
| Stable tool interface | Decouple reasoning from execution | High | Tool semantics or protocol contract changes |
| Forced task decomposition | Compensate for weak long-horizon planning | Variable | Model planning or context behavior improves |
| Context reset or compaction rule | Prevent late-task degradation | Variable | Context window, retrieval, or persistence changes |
| Model-based evaluator | Detect defects the worker misses | Variable | Worker or evaluator model changes; task mix shifts |
| Fixed retry count | Raise success probability | Low to variable | First-pass reliability, cost, or consequence changes |
None of these labels is permanent. Even a durable control can be implemented badly, and even a temporary workaround can remain necessary for years. The table identifies the kind of evidence each one needs.
Scaffolding debt is paid on every run
Ordinary technical debt often waits quietly until someone changes the affected code. Scaffolding debt is more active. It sits in the execution path.
An obsolete planner adds tokens and delay to every task. A stale decomposition rule can split work at the wrong boundaries. An unnecessary evaluator increases cost and may reject correct but unfamiliar solutions. A context transformation can discard information a stronger model would have used. A retry rule can turn a rare mistake into repeated side effects.
The cost is not only inference spend. More components create more states to observe, more failures to classify, and more interactions that can change during a model upgrade. The orchestration layer can make the system harder to reason about precisely when its purpose was to make the model easier to trust.
Complexity also creates attribution errors. If a workflow improves after a model release, the model gets credit even when an orchestration change caused the gain. If it regresses, the model gets blamed even when an old grader or context rule constrained it. Without component-level evidence, the team cannot tell whether it is buying model capability or paying orchestration tax.
This problem becomes acute when one control loop is shared across task classes. A rule introduced for one difficult workflow quietly governs every easier one. The exceptional case becomes the default architecture.
Give every compensating component an expiration trigger
Teams already know how to manage temporary architecture. Feature flags have owners. Database migrations have rollback plans. Compatibility layers are linked to the clients that still require them. Agent-specific orchestration needs similar discipline.
For every compensating component, record five things:
- The observed failure it addresses.
- The task segment and model version on which that failure was measured.
- The outcome metric it improves, including cost and latency.
- The evidence that would justify removal.
- The safe rollback path if removal regresses performance.
The record prevents a planner from becoming folklore.
The record can be compact. A row in a versioned assumption manifest may be enough:
| Component | Assumption | Evidence | Expiration trigger | Removal check |
|---|---|---|---|---|
| Per-phase evaluator | Worker misses cross-phase integration defects | 14 of 60 replay cases repaired on model A | Worker model or task architecture changes | Compare accepted outcomes, cost, and defect escape rate with evaluator on and off |
The numbers above are illustrative, not a benchmark. Their purpose is to make the team's causal claim visible.
When a model or major tool changes, run ablations against a deployment-shaped replay set. Remove one compensating component at a time. Compare not only task completion but accepted outcome quality, latency, inference cost, retries, and the kinds of defects that escape. Restore the component if current evidence still supports it.
This is more reliable than radically simplifying the whole system at once. Anthropic reached the same conclusion in its long-running application work: removing several pieces together made it difficult to identify which parts were load-bearing. Methodical removal produced usable evidence.
A model upgrade is also an orchestration migration
A model upgrade can change when the system asks for clarification, how it selects tools, how much context it can use, whether it follows a forced plan too literally, and which defects a second model can detect. Because the orchestration layer encodes assumptions about those behaviors, changing the model is a migration of the pair, not an identifier update.
Release the model and its runtime as a tested pair. Preserve the previous pair for rollback. If a new model permits a simpler system, treat deletion as part of the upgrade rather than an optional cleanup phase that never arrives.
The strongest version of this practice creates two different rates of change. Stable interfaces for sessions, tools, execution, and evidence evolve cautiously. Model-specific choreography is explicitly versioned and expected to turn over faster. Anthropic's Managed Agents design uses this separation: durable sessions and replaceable control loops can fail or change independently.
That architecture accepts a reality many agent systems still resist. The model is not the only replaceable part. The reasoning loop around it is replaceable, too.
The best orchestration layer remembers why it exists
There is no prize for running an agent with the fewest components. There is also no safety in preserving complexity whose justification has expired.
A good agent runtime makes difficult work more dependable. A maintainable one can also explain which difficulty each component addresses, what evidence supports it, and when the team will reconsider it.
That is the difference between infrastructure and sediment.
As models improve, some scaffolding will become unnecessary. Other controls will become more important because the system can act farther, faster, and with more authority. The job is not to predict which category every component belongs to forever.
The job is to keep the distinction testable.