Hallucinations

invented libraries, methods and parameters

What it is

A hallucination is when the AI confidently describes something that does not exist: a library, a method, a parameter, a field in an API response. It looks like ordinary code, reads like ordinary code, and fails when you run it.

The phenomenon is not random but predictable: it appears where the examples were scarce — rare libraries, recent versions, narrow APIs (article 1.3).

What it looks like in code

async with aiohttp.ClientSession() as session:
    data = await session.fetch_json(url)      # no such method

Plausible: requests has .json(), plenty of clients have fetch. The combination looks natural — and does not exist.

Three kinds, in descending order of visibility:

The first and the third are more dangerous than they look.

The package. The names an AI invents repeat from answer to answer, and people register them in advance on PyPI and npm — with their own code inside. Then pip install succeeds, the import works, and the package does whatever was put in it. "The package exists" does not mean "it is the right package".

The parameter. A Python function complains about an unknown argument straight away:

await session.get(url, timeout_ms=5000)

TypeError: ClientSession._request() got an unexpected keyword argument 'timeout_ms'. Did you mean 'timeout'?

It passes silently where nobody checks parameters: in a JavaScript options object ({ timeoutMs: 5000 }), in **kwargs passed on unchecked, in YAML and JSON configs. No error, the default applies, and the behaviour is not what you intended.

How to check

A package:

pip index versions aiohttp

If the package does not exist, the command says so. It also shows the available versions.

That is only half the check. The other half is whether it is the right package: open its page on pypi.org or npmjs.com and look at how old it is, how many downloads it has and whether it links to its source. A package created last week under a name the AI suggested is a reason to stop.

A method or a field:

python3 -c "import aiohttp; print([m for m in dir(aiohttp.ClientSession) if 'fetch' in m])"

[]

An empty list — the method isn't there.

A function's parameters:

python3 -c "import inspect, aiohttp; print(inspect.signature(aiohttp.ClientSession.get))"

(self, url: Union[str, yarl.URL], *, allow_redirects: bool = True, **kwargs: Any)

The signature shows which arguments are accepted. But when it contains **kwargs, as here, it proves nothing: the arguments are passed further along, and only the documentation or a call with that parameter will tell.

All three checks are one line in the terminal, and they are more reliable than any amount of reasoning.

What to tell the AI

Warning is cheaper than correcting:

Only use methods you are sure exist.
If you are not sure, mark the line with a # verify comment
and say exactly what to check in the documentation.

This wording works: by explicitly allowing doubt you get honest markers instead of confident invention.

After the code arrives:

List every external method and parameter you used in this code,
with the library and version. As a list, no code.

A short list is convenient to check in a minute — faster than rereading the code.

When it has already failed:

AttributeError: 'ClientSession' object has no attribute
'fetch_json'. I have aiohttp 3.9.5. Which method does exist
there for this job?

What to look for

A suspiciously convenient method. If a library offers exactly the function you need in a single call, check it. Real libraries rarely match a task that precisely.

An unusual name. fetch_json, get_all, auto_retry — names that sound like what ought to exist. That is the sign: the AI assembled the name from parts it saw in similar libraries.

Parameters not in the documentation. The least visible kind. If the parameter matters, check the signature.

The code was never run. Running it catches invented methods and almost all invented parameters. A package whose name someone has already taken is not caught by running. Running it is the minimum defence, and it is mandatory.

A new package in the dependencies. The AI added a library you did not ask for — check it before installing it, not after.

Invention in the API response, not in the code. The AI may assume which fields will arrive in the JSON and be wrong. Print the real response rather than trusting the description (article 4.5).

Cheat sheet