============================================================ nat.io // BLOG POST ============================================================ TITLE: AI Code Has a Conversation Debt DATE: July 11, 2026 AUTHOR: Nat Currier TAGS: AI Systems, Software Engineering, Engineering Leadership, Developer Experience ------------------------------------------------------------ The most dangerous AI code is not the code that fails loudly. It is the code that works well enough to end the conversation. When a junior engineer makes a questionable choice, a good reviewer asks why. The answer exposes a model: an assumption about latency, a missing edge case, an incorrect constraint, or a tradeoff the engineer has not yet learned to see. The reviewer challenges it. The engineer adjusts it. Both leave with a better map of the system. Fully AI-generated code can skip that exchange. A developer may know which prompt produced the patch but not why the cache key is safe, why the retry boundary lives where it does, or what failure mode the validation is actually protecting. When the code later misses something, the team does not refine an understandable line of reasoning. It reverse engineers a plausible artifact. That is **conversation debt**: the accumulated cost of changes whose reasoning was never made available for review, challenge, and learning. It is not a complaint about junior developers taking the easy path. Most people will take the faster path when deadlines, incentives, and tooling reward it. The failure is organizational. Teams adopted a production accelerator without redesigning the feedback loop that turns implementation into judgment. > **Thesis:** AI-generated code becomes risky when it removes the human reasoning conversation that review is supposed to improve. > **Why now:** Agentic tools can create more implementation output than a team can meaningfully understand through conventional review habits. > **Who should care:** Engineering leaders, senior developers, security reviewers, and developers learning to work with AI every day. > **Bottom line:** Do not ask people to explain every generated line. Require them to own the decisions, assumptions, and verification story that make the change safe. [ Key Ideas ] ------------------------------------------------------------ - Review is not only a defect filter. It is the place where an engineer's mental model gets upgraded. - AI can create a gap between code ownership and reasoning ownership. That gap compounds into quality, security, and dependency risk. - The answer is not AI prohibition. It is a new contract for decision traceability, explanation, and deliberate learning. [ A passing diff can still contain a missing conversation ] ----------------------------------------------------------------- The following is a composite drawn from a pattern I keep seeing in AI-assisted reviews: a junior developer is asked to prevent duplicate charges. An AI agent proposes an idempotency key, a database check, a retry wrapper, and tests. The pull request is concise. The tests pass. A reviewer sees reasonable code and approves it. Weeks later, an incident reveals that the key is derived before a customer-visible order state exists. Retries avoid duplicate provider calls in one path while allowing a different duplicate path through a background worker. Nobody chose that tradeoff. Nobody can explain why the boundary was placed there. The code is not the work product of a reasoning process the team can iterate on. It is a starting point for archaeology. This is not a claim that humans never write opaque code. They do. The difference is that a human author usually carries some recoverable causal context, even when it is incomplete. A conversation can surface it. With AI-first implementation, the author may have outsourced the causal model along with the syntax. > **Review signal:** If an engineer cannot explain the assumption a generated change depends on, the team has not finished the review. It has only inspected the artifact. [ The feedback loop is how people become reliable ] ------------------------------------------------------------ Engineering judgment is not a library of rules someone memorizes before they become senior. It is compound interest on challenged decisions. A developer suggests an approach, gets asked what happens under load or retry or partial failure, and discovers that their initial model was too small. The next time, they notice the boundary earlier. That learning loop matters because code review has two outputs. The obvious output is a safer change. The less visible output is a stronger engineer. Teams that measure only merge speed will miss the second output until they discover that nobody can diagnose a failure without asking the model that produced the patch. AI changes the economics of this loop. It can make a developer appear productive before the developer has formed the mental models that normally make productivity durable. The team gets a feature. The developer gets fewer repetitions of explaining, being challenged, revising, and connecting a local choice to a system consequence. That is not laziness. It is a predictable response to a workflow where generation is immediate and reflection is optional. The problem becomes acute when the developer is overworked. If a deadline rewards output and does not protect time for explanation, "use AI for everything" is often the rational local choice. Leaders should not moralize about that choice. They should change the environment so fast generation still produces a learning artifact. [ Conversation debt compounds in three directions ] ------------------------------------------------------------ The first direction is **quality**. Missing reasoning makes it harder to distinguish intentional tradeoffs from accidental behavior. Reviewers check whether code looks conventional instead of testing whether the decision is right for this system. The second is **security**. Security failures often live in assumptions: which identity is trusted, which input is canonical, which state transition is safe, which failure should deny rather than retry. A generated patch can satisfy a narrow feature request while carrying an unsafe assumption from a context it never saw. The third is **dependency**. If a team cannot reproduce the reasoning behind its own changes, every future modification depends more heavily on external model output. The dependency is not just on a tool. It is on a tool becoming the only participant that can rapidly generate a plausible explanation. This is why conversation debt is not merely documentation debt. Documentation can describe what happened. Conversation develops the ability to decide what should happen next. | Weak AI workflow | Learning-preserving AI workflow | | --- | --- | | Prompt, patch, approve | Prompt, decision brief, patch, challenge, verify | | Reviewer reads code for plausibility | Reviewer tests assumptions and boundary choices | | Author owns the output | Author owns the claim and can explain it | | Failure triggers reconstruction | Failure updates a visible decision record | [ The new review contract should target decisions, not keystrokes ] ------------------------------------------------------------------------- Nobody needs a ceremony in which a developer narrates every line an agent produced. That would make AI assistance unusable. The contract should be narrower and more valuable: the developer must state the decision the change makes, the assumptions it depends on, the failure condition that would falsify it, and the proof that was run. For the duplicate-charge example, a review-ready note might say: "We create the idempotency key after the order reaches pending payment, use the same key across synchronous and worker retries, and reject a second provider call. The focused tests cover both paths. The remaining risk is an old event without a stable order ID, which stays on the existing reconciliation path." That is a conversation starter. A reviewer can now challenge the actual boundary. Make these fields small enough to be used: **decision**, **assumption**, **failure mode**, and **proof**. AI can draft them. The human owner should attest that they are true. This turns generated code from a black-box answer into a bounded engineering claim. ```mermaid flowchart LR A[Task and context] --> B[AI proposes implementation] B --> C[Developer states decision and assumptions] C --> D[Review challenges boundary] D --> E[Focused verification] E --> F[Commit and learning record] F --> G[Future change starts with visible reasoning] ``` At this point, the goal is not perfect transparency. It is a feedback loop strong enough that the next decision begins from better judgment than the last one. [ A review note can make reasoning inspectable ] ------------------------------------------------------------ The change does not need a long design document. It needs a record that lets a reviewer challenge the reason for the change rather than reverse engineer it from generated syntax. **Before input: a merge-ready-looking request** ```text Prompt: prevent duplicate charges; add retries and tests. PR description: Fixed duplicate payments. Tests pass. ``` **After output: the decision record a reviewer can challenge** ```text Decision: one payment attempt per order in pending_payment. Assumption: every retry path has the persisted order ID. Failure mode: legacy events lack that ID; route those to reconciliation. Proof: tests cover HTTP retry + worker replay + provider timeout. Owner: payment team; rollback: disable the new worker path. ``` The first artifact gives a reviewer code. The second gives them a falsifiable claim. The operational result is sharper review: a reviewer can reject the missing worker invariant instead of rereading generated syntax for a familiar pattern. AI can draft the record, but it cannot truthfully attest to a local invariant it was never accountable for. [ Review quality is a leading indicator, not an after-action metric ] --------------------------------------------------------------------------- Teams should sample whether changes contain a decision, assumption, failure mode, and proof, then use incidents to discover which missing field mattered. They should not turn this into individual surveillance or a ritual score. The point is to see whether the team is building recoverable reasoning as generation accelerates. So far, conversation debt is the gap between code output and recoverable reasoning. The duplicate-charge path shows the next control: make that reasoning small enough to survive delivery pressure and precise enough to challenge before the code becomes production history. [ Senior engineers must review the model, not only the patch ] -------------------------------------------------------------------- The senior role changes here. It is no longer sufficient to be the person who can spot style problems or rewrite a difficult function. The high-leverage senior engineer asks the questions that turn a generated patch into an understood decision: What state does this assume? What happens when the request repeats? Which invariant is this protecting? What must be true for the test to be meaningful? That is mentorship with an operational output. It protects the current change and trains the developer to make the next AI interaction better. The most useful review comment is often not "change this line." It is "what would make this approach wrong?" Teams should also create a protected space for explanation. Rotate short design reviews for agent-generated changes in high-risk areas. Let developers bring a decision they do not yet understand. Reward a pull request that becomes smaller after a reviewer finds an assumption. Those moments are not evidence that AI failed. They are evidence that the human learning system still works. [ The point is not to make AI slower ] ------------------------------------------------------------ The right response to conversation debt is not to demand that every developer hand-write boilerplate to prove they deserve AI. That would confuse friction with learning. Use AI to accelerate implementation, generate test cases, surface alternatives, and summarize context. Then preserve the places where judgment is built: naming the decision, defending an assumption, testing the boundary, and learning from the result. The durable team will not be the one that produces the most code with AI. It will be the one that converts AI speed into human capability rather than using speed to hide a missing feedback loop. If you lead, review, or learn inside an AI-assisted team, begin with one high-risk path and introduce the four-field record there. The goal is a small review contract, not a transcript of how the code was generated. When the duplicate-charge path next changes, the team should inherit an argument it can improve rather than a patch it has to decode. **Code can be generated. Judgment has to be practiced in public.**