The illusion of finished code

why "it works" does not mean "it's right"

What it is

Code from an AI looks finished. Even indentation, sensible names, type annotations, tidy comments. It looks like code somebody thought about.

That impression is deceptive, and it is deceptive for three reasons. They are worth taking apart at the start of the book, because everything that follows is built on not falling for them.

Three illusions

Illusion one: it looks tidy

How neat code is says nothing about whether it is right. A person who writes sloppily usually writes sloppily everywhere, so the look of the code told you something. With an AI that link is gone: it formats a correct solution and a wrong one with exactly the same care.

Worse, tidiness works against you — it lulls. Code with crooked indentation gets read carefully; code that looks like a textbook gets scrolled past.

Illusion two: it ran

A run checks exactly one case — the one you ran. Usually that is the happy path: the city was found, the data arrived, the user typed what was expected.

temp = data.get("main", {}).get("temp", 0)

This line will always run. On correct data it gives the right answer; on a server error it gives "It is currently 0 degrees". The program works. The program lies.

That is exactly why this book pays so much attention to the unhappy cases: empty data, dropped connections, garbage input. That is where the errors live that a run does not find.

Illusion three: the AI sounds sure

The tone of an answer is unrelated to how right the AI is (article 1.3). It does not say "I'm not sure", does not hedge, and does not mark the places where it was guessing. An explanation of a method that does not exist sounds every bit as convincing as a description of one it knows cold.

What to use instead

The book proposes replacing the feeling of "seems fine" with four concrete questions. They are the same questions as in article 2.1, and they close all three illusions:

1. What does this function take and what does it return?
2. Does it change anything outside itself?
3. What happens if the data is not what was expected?
4. Where is it called from?

The third question is the antidote to the second illusion. It forces you to think about the cases you did not run.

What to tell the AI

The illusion of confidence is broken by a direct question:

Name three places in this code where it will break
on unexpected data. If there are none, explain why not.

The illusion of the run — by a question about what went unchecked:

I ran it and got the right answer. Which cases did
my run not check?

And the illusion of tidiness — by asking for the trade-offs:

Which decisions in this code are debatable? What did you
choose without thinking, because it is the usual way?

The last question often pulls out the most interesting material: decisions the AI made out of habit rather than because they fit your case.

What to look for

You accepted the code without asking a single question. Normal work includes a few clarifications on every non-trivial change.

Checking came down to running it. Ran it, answer looked right, accepted. That checks one scenario out of ten.

Comments instead of code. # TODO: handle the empty list is a hole, dressed up to look like it was considered (article 2.14).

The feeling that it all went too smoothly. If the task was non-trivial and the solution arrived without a single caveat or question, part of the difficulty has probably just gone unnoticed.

Cheat sheet