A language model is trained to produce fluent, plausible continuations, not to independently verify correctness. So it produces wrong code in the same style as correct code — a citation to a nonexistent function, an off-by-one in otherwise clean logic — which is why a casual read misses it. The fix is checking ground truth, not re-reading.
Key takeaways
- Fluency and correctness are different properties. A model optimized to produce plausible text produces wrong code that reads exactly as well-formed as right code.
- The failure mode has a name in the literature: hallucination — confident, well-formed output that isn't grounded in fact, whether that's a citation, an API call, or a logical step.
- Self-review by the same model helps partially, not fully — a confident-sounding confirmation from the same generative process isn't independent verification against ground truth.
- What actually catches it: execution, not inspection. Running the test suite, a type checker, a linter, or the code itself catches errors a visual read misses, because these check against reality instead of plausibility.
- The riskiest code to accept unverified is the kind that looks the most polished — well-named variables and idiomatic structure make an error harder to spot, not easier.
Why fluent and correct aren't the same thing
A large language model generates the next token by predicting what's statistically plausible given everything before it, trained on enormous volumes of code that mostly demonstrates correct patterns. That training produces genuinely strong pattern-matching: idiomatic naming, sensible structure, the right library calls in the common case. But plausibility and correctness are different properties measured differently — one asks "does this look like code that would appear here," the other asks "does this actually do what it claims." A model can be extremely good at the first and still wrong on the second for a specific instance, because nothing in how it generates text forces the two to agree.
| Failure type | What it looks like | Caught by a casual read? |
|---|---|---|
| Typical human bug | Awkward structure, obvious shortcut, missing handling that stands out visually | Often, yes |
| Fabricated API/library call | A method or parameter that doesn't exist but follows real naming conventions | Rarely — needs checking against the actual installed version |
| Subtly wrong logic in clean code | An off-by-one, wrong comparison, or unhandled edge case inside otherwise idiomatic code | Rarely — the surrounding polish reads as a signal of correctness it doesn't actually provide |
Why asking the AI to check its own work only partially helps
Prompting a model to double-check its own output can catch some errors — it's a genuinely useful step — but it shares the same underlying mechanism as the original generation. A confident-sounding "yes, this looks correct" from the same generative process is not independent verification; it's another plausible continuation, this time in the shape of a confirmation. It's not nothing, but it isn't grounded in the same way running the actual code is.
What actually catches it
- Run the test suite — a passing test checks behavior against a defined expectation, not against how plausible the code looks.
- Use a type checker or linter — catches an entire class of errors (wrong types, undefined references, unreachable code) mechanically, without depending on a reviewer noticing.
- Actually execute the code against real or representative input, especially for anything with edge cases — empty inputs, boundary values, concurrent access.
- Verify cited APIs and library calls exist in the installed version being used — a fabricated method name is invisible to a reviewer who doesn't have the library's actual interface memorized.
- Treat well-polished output with the same scrutiny as rough output, not less — polish is not evidence of correctness for AI-generated code the way it can be a weak signal for human-written code.
Documented vs. inferred vs. unsupported
| Claim | Status | Basis |
|---|---|---|
| LLMs can generate fluent, confident output that is factually or logically wrong | Documented | Widely observed and discussed failure mode across LLM applications, commonly termed "hallucination" |
| Execution-based checks (tests, type checkers) catch errors a visual review misses | Documented | Standard software engineering practice, independent of whether the code was AI- or human-written |
| AI-generated code is always less reliable than human-written code | Unsupported | Depends heavily on task type, model, and verification process used — not a fixed, universal ranking |
| A specific percentage of AI-generated code contains bugs | Unsupported | Varies enormously by task, language, and codebase — figures circulated online rarely specify a reproducible measurement method |