Tests — read them and run them

what red output means and what it actually believes

What it is

How to ask for tests is article 3.9. This article is about what to do with the ones you got: how to read pytest output, what counts as a real check, and why green tests do not mean everything is fine.

You will not have to write tests yourself. You will definitely have to read them — otherwise you are trusting a check you never checked.

What it looks like in code

A function splits a long message into chunks. The tests were written from the list of cases in article 3.9.

pytest -q

...F                                                                     [100%]
=================================== FAILURES ===================================
_______________________ test_paragraph_longer_than_limit _______________________

    def test_paragraph_longer_than_limit():
        text = "y" * 300
        parts = split_message(text, limit=100)
>       assert all(len(p) <= 100 for p in parts)
E       assert False

test_split.py:23: AssertionError
=========================== short test summary info ============================
FAILED test_split.py::test_paragraph_longer_than_limit - assert False
1 failed, 3 passed in 0.02s

How to read it

The first line is the map. ...F means three tests passed and the fourth failed. A dot is a pass, F a failure, E an error in the test itself, s skipped.

> marks the line it failed on. Here, the assert.

E says what didn't match. assert False means the condition turned out false.

The last line is the bottom line. 1 failed, 3 passed is the most important part, and a fine place to start.

What the failing test showed

The function cannot handle a paragraph that is itself longer than the limit: it can't cut it and hands it back as is. The other three tests are green — on ordinary data everything works.

This is exactly the case marked in article 3.9 as "the one people usually forget". The test caught it — which is the whole point.

Useful flags

pytest -q                       # brief
pytest -x                       # stop at the first failure
pytest -k "split"               # only tests with "split" in the name
pytest test_split.py::test_empty  # exactly one test
pytest -q --tb=short            # a short traceback

The second and fourth are the workhorses while fixing: no need to wait for the whole suite each time.

What to tell the AI

Here is the pytest output. What exactly didn't match and why?
Don't fix the code yet.

<output>

When a test fails after your change:

This test passed before my change and fails after.
I only changed function X. What in it could cause that?

And a quality check on the tests themselves:

Do these tests really check behaviour, or do they check that
the code equals itself? Are any of them going to pass against
any implementation at all?

What to look for

A test that cannot fail. The check: break the code deliberately and run it. If the test stays green, it checks nothing.

assert result is not None where None is impossible. Looks like a check, checks nothing.

Mocks instead of checking. If half the system is replaced with stubs, the test verifies that the stub returned what was put into it.

A test that repeats the implementation. If the test reproduces the function's algorithm line by line, it will break on any refactoring, even a correct one.

All cases happy. Not a single test with empty data, None or garbage input — the set is incomplete, and that is exactly where bugs live.

Green tests instead of reading the code. Tests check what is written in them and nothing more. They complement reading the diff; they do not replace it.

Cheat sheet