Agent-written tests and mutation testing: an argument not yet proven

In the first article in this series I argued that agents move development cost upstream to specification and downstream to verification. I described two opposing agents, language models autonomously using development tools: one writes tests from the specification, the other writes code, with mutation testing measuring whether small code changes make tests fail. What was missing was evidence. This is the test.

The conclusions differ somewhat from what I expected. Three points from section 5 hold up: full coverage can hide a blind test; raw scores mislead; unrequested code is an exposed area. But the stronger argument, that separating agents reveals gaps, is not proven. And the most important signal did not come from the tests at all.

These results are not conclusive. Two small features, precise specifications, the same model for both agents, and classifications made by models rather than a person. More intensive testing, with more features, replications, providers and less polished specifications, is needed before drawing general conclusions. This is an honest experiment rather than a demonstration.

Who did what

This changes the weight of everything else: AI agents performed all technical work in a Claude Code session on Taita, my side project: FastAPI backend, PostgreSQL, Python 3.11 and about 2,300 tests, in October 2026. As author, I performed no manual checks. I provided direction and asked the critical questions reproduced here.

Role	Who
Specifications, orchestration, integration, mutant classification, handwritten mutants, CI analysis and corrections	Orchestrator agent, Claude Code
Tests written only from the specification	Test agent, a sub-agent in an isolated Git worktree, a separate checkout of the repository
Code written only from the specification	Code agent, a sub-agent in an isolated worktree
Independent verification of failed tests	Reviewer, a sub-agent using a different model, Sonnet, without knowing the orchestrator’s conclusions

Classifications, diagnoses and judgements are model outputs, not human assessments. An agent wrote the specifications too.

My instructions and questions, in order:

1. Run the experiment on Taita.
2. “If a test involving mocks and a database seems useful, do it.” Mocks substitute dependencies, here the notification service, to isolate the code.
3. Have another model check whether failing tests were themselves faulty.
4. Prepare the PR with pipefail and fix broken tests.
5. “What is the point of testing on SQLite?” / “Why are we still using SQLite?”
6. “If you are using Postgres, does rerunning mutation testing make sense?”
7. “Could one of the mutants timing out run in parallel?”
8. “Does running all the tests make sense?” → “Change conftest.” → “That does not seem like much improvement.”
9. “Did it find a real issue and demonstrate that opposing agents help?” → “Then let us try to demonstrate the argument.”
10. After 013: “What if you used agents with different characteristics rather than different models?” → the reduced 014 version was selected because of credit limits.

For experiments 012–014, the specifications, hidden intent, reference implementations, seeded bugs, prompts, decision rules and classifications all came from the orchestrator.

Part 1: do agent-written tests hold up to mutation testing?

Protocol, identical for two features

1. The orchestrator wrote and committed the specification before any code.
2. Two sub-agents started in parallel in separate Git worktrees, without shared context:
   • Test agent: only the specification, with no implementation yet; a neutral prompt requesting a complete, merge-ready suite, without mentioning mutation testing.
   • Code agent: only the specification, explicitly forbidden to write tests.
3. Integration → coverage, including exercised branches → mutmut 3.8.0, mutating only the new module and running only its suite.
4. Second iteration: the test agent received the raw survivor report, without equivalent mutants identified, and was asked to kill only real specification gaps and justify the rest.

Feature 1: pure function, school days and bridge holidays

• About 180 lines, no I/O. 396 test cases, all green on the first run against the other agent’s code. 100% line and branch coverage.
• mutmut round 1: 87/93 killed, 93.5%, about 17 seconds.
• Six survivors: four error-message text changes outside the contract; one equivalent mutant, changing < to <= but reassigning the same value; one real gap.
• The real gap: a test called test_default_max_length_is_one. Correct name, green, full coverage, but its chosen data produced the same result for max_length=1 and max_length=2, so it could not detect the wrong default. Only a mutant changing the default from 1 to 2 exposed it.
• Round 2: the test agent killed that mutant, independently recognised the equivalent one and justified leaving the four message mutants alive. 88/93, no specification gaps.
• The agent spontaneously built an oracle, a second implementation derived from the specification, and compared it with the code across forty random school years. About 930 test lines for 180 implementation lines.

Feature 2: database plus mocks, family bridge-holiday notifications

• Reads Family and SchoolCalendarEvent through an ORM, mapping database rows to language objects, uses feature 1 and sends notifications through a mocked asynchronous service.
• 93 tests, 91 green initially. The two failures were false positives from the test agent: checks for “one database query” counted an extra query caused by the test itself reading family.id after expire_all(). The orchestrator corrected them, the only test intervention. The independent reviewer confirmed the diagnosis and that the correction did not weaken the tests: injecting another query into the implementation made them fail.
• 97% branch coverage. Uncovered lines were outside the specification, added unprompted by the code agent.
• mutmut round 1 on SQLite: 161/188, 85.6%, about 5.5 minutes. The orchestrator classified 27 survivors before seeing the test agent’s judgement:
  • Thirteen logging text or argument changes; the specification only prohibited personal data in logs.
  • Eight in unrequested code.
  • Six equivalents, one exposing a specification defect, an impossible multiline message.
  • No real specification gaps.
• Round 2: the agent agreed on all 27. It killed one mutant, continue to break, explicitly stating that the new test fixed unspecified behaviour. 164/188. Two of the three additional kills happened accidentally, from invalid logging format causing errors rather than assertions.

	Pure function	Database plus mocks
Green on first run	396/396	91/93, two faulty tests
Branch coverage, round 1	100%	97%
Mutation score, killed mutants divided by total, round 1	93.5%, 87/93	85.6%, 161/188
Mutation score, round 2	94.6%, 88/93	87.2%, 164/188
Real specification gaps, round 1 → 2	1 → 0	0 → 0

Part 2: green CI that could not fail

Here is the twist, and why this article is no longer called “coverage and blind tests”. The independent reviewer concluded an existing test_reactivation “probably also fails in CI”, the automatic build-and-test pipeline, yet CI was green. To resolve that contradiction, the orchestrator read the logs rather than trust the status.

• The pytest step was python -m pytest ... 2>&1 | tee pytest-output.txt, without pipefail, the shell option that propagates upstream pipeline failures. Without explicit shell: bash, GitHub Actions runs scripts with bash -e. A pipeline returns the exit code of its last command, tee, which copies output to a file and normally succeeds. No failed test could fail the job. The mypy static type-checking step had the same defect and could never block, even with mypy-blocking: true.
• Git history showed the line existed from at least 7 August 2026.
• The latest “green” Postgres run on main logged: 5 failed, 1735 passed, 28 skipped, 16 deselected, 83 errors.
• Backend deployment invoked the same checks, so deployments also passed with red tests.
• The AI itself had trusted the green status: the orchestrator had told me “CI is green, the tests pass on Postgres too”, inferred from the indicator. An unverified claim, corrected after reading logs.

What it concealed

• 83 errors, one cause: thirteen test files created their own in-memory SQLite databases, and the shared client fixture, code preparing and cleaning test state, was always SQLite, even in the “Postgres” run. Those tests had never run on Postgres.
• Five failures, all test bugs, none in production code:
  • Violated foreign keys, relational integrity constraints, involving reminders, family members and Gmail integrations. SQLite does not enforce them by default.
  • Handwritten SQL missing the schema.
  • A time assertion valid only on SQLite, which stores local time without a timezone, while Postgres returns timestamptz in UTC.
  • A test not simulating the scheduler’s database role. The implementation correctly refused cross-family queries without that role.
• A sixth test, test_reactivation, failed only locally because it called the real email service outside the mock. The independent reviewer diagnosed it.

The correction

The orchestrator initially moved those thirteen files into the repository’s “SQLite-only” list. That made CI honest but preserved the blind spot. After I asked “what is the point of SQLite tests?”, it migrated them to Postgres and made client Postgres-aware.

CI run	Before	After
Postgres	5 failed, 83 errors, 1,735 passed: green job	1,823 passed, none failed: green with pipefail
SQLite, historical list	466 passed	466 passed

No production code changed. Every issue was in tests or CI.

Why SQLite was still there

Inertia: it is the local default, needs no running database, about thirty files had never migrated, and it is fast and easy to parallelise. Here it concealed exactly the issues discovered: foreign keys, timezones and schemas. It also cannot verify Row-Level Security (RLS), filtering visible rows per user, BYPASSRLS, the PostgreSQL role attribute bypassing those policies, or row ordering: central security guarantees in this project.

Part 3: the same mutation testing on Postgres

My question: “If you are using Postgres, does rerunning mutation testing make sense?” Yes for database-dependent feature 2; no for pure-function feature 1.

Same tests and code, different database. mutmut used one process on Postgres. Eight processes would share a database, interfering and producing mutants “killed” by collisions rather than effective tests.

First run: misleading numbers

	SQLite, eight processes	Postgres, one process
Killed	164	152
Survived	24	0
Timed out	0	36
Duration	5 minutes	92 minutes

At first glance, Postgres says “zero survivors”: perfect tests. That is false.

Why the timeouts happened

• mutmut allows each mutant (estimated test time + 1) × 15. For the feature’s 95 tests it estimated 4.2 seconds, giving a 78-second limit.
• The actual Postgres test file took about 220 seconds, because every test created and dropped tables. The figures suggest mutmut’s estimate omitted this cost. That is inferred from data, not verified in mutmut’s code.
• mutmut runs tests shortest estimated duration first, confirmed in code and stored statistics. The test_scenario1... test killing several mutants was 94th of 95 in that order, but first in normal pytest order.
• Both real survivors, needing the full suite, and real kills whose detecting test came late therefore timed out.

Checking the 36 timeouts

The orchestrator reran them without a time limit directly against mutmut’s mutants using MUTANT_UNDER_TEST, four in parallel, each with its own cloned Postgres database, my suggestion.

All 36 checked: 23 real survivors, 13 real kills. None differed from SQLite.

• Hidden survivor: mutant 20, closure name to None, passed 95 tests in 227 seconds; mutmut called it a timeout.
• Hidden kill: mutant 50, max_length 1 to 2, failed one test, but after about 225 seconds; mutmut called it a timeout.
• Six mutants died in 11 seconds in pytest’s first test, but timed out in mutmut because of ordering.
• One false kill: mutant 11, removing a logger.warning argument, was reported killed on Postgres. Rerun directly, it survived in both pytest and mutmut order: 95 passed in 247 seconds. The cause was not established.

The verified Postgres result was 164 killed and 24 survivors, identical to SQLite. Postgres changed how the tool reported results, rather than which mutants the tests detected.

The real cause: slow fixtures

I asked whether running all tests made sense. Only the feature’s 95 tests were running already, and all exercised its single entry point, so all were needed. The problem was the test_db fixture, creating and dropping about 110 tables for every test, roughly two seconds per Postgres test.

I requested a change, implemented by the orchestrator:

• First attempt: create the schema once, then TRUNCATE, quickly emptying tables, after each test. The file dropped only from about 220 to 130 seconds. I said that was not much improvement. Profiling showed truncating about 110 tables still cost 1.5 seconds per test, a fixed per-table cost.
• Second attempt: per-table DELETE with foreign-key constraints suspended in the transaction, plus sequence resets. About 220 seconds became 23 seconds for the feature file.

Across the Postgres suite, local time fell from 759 to 146 seconds, with all 1,827 tests green. The GitHub CI Postgres pytest step fell from 562 to 116 seconds, with all 1,823 tests green.

Second Postgres mutmut run with fast fixtures

Same defaults, one process, no direct manual checking.

mutmut, feature 2	SQLite	Postgres, slow fixtures	Postgres, fast fixtures
Killed	164	152, including one false kill	164
Survived	24	0	24, the same individual mutants as SQLite
Timed out	0	36	0
Duration	5 minutes, eight processes	92 minutes, one process	19 minutes, one process

With fast fixtures, mutmut reported the true Postgres result exactly. Mutant 11, the earlier false kill, correctly survived too. The anomaly belonged to the slow run. Postgres itself was not the issue: fixture cost combined with mutmut’s estimates and ordering was.

Lesson

On a real database with slow fixtures and default mutmut settings, “timeout” tells us nothing in either direction. Counting timeouts as detected, a common convention, inflates the score to 100%; counting them as failures depresses it. Fast fixtures eliminated the problem: zero timeouts, identical SQLite results and nineteen minutes instead of ninety-two. The cost and reliability of database mutation testing depend primarily on fixture design.

Part 4: opposing agents, does separation actually help?

Did these results show that a separate testing agent finds problems a single agent would conceal? The honest answer was no. Three preregistered experiments followed: the orchestrator fixed protocol, decision rule, hidden intent and reference implementation before any agent ran, committing SHA-256 hashes to seal them against invisible retrospective changes. All agents used the same model unless stated otherwise, with three replications per condition.

012: precise specification, school-run rotation

Conditions: A, separate testing and coding agents; B, one agent writes both; C, like A but with a different testing model, Sonnet. Measurement: fifteen orchestrator-seeded bugs, verified non-equivalent, plus real bugs in agent implementations.

Suite	Seeded bugs detected	False positives
A1/A2/A3	15/15/15	0
B1/B2/B3	15/14/15	0
C1/C2/C3	15/15/15	1/0/0

All six implementations matched the reference across 6,000 random cases each. Inconclusive: a ceiling effect, where near-perfect scores prevent distinguishing conditions. With a precise specification, everyone wrote correct code and tests detecting everything. Eight of nine suites contained a second specification implementation as an oracle, including suites written by the coding agent itself.

013: vague specification, splitting family expenses

The specification was deliberately brief: “split fairly”, “few transfers”. Ten product owner decisions remained hidden. Conditions A and B were as above.

• All six implementations diverged from intended behaviour, avoiding a ceiling. Four diverged mildly, settling equal-amount pairs first and changing transfer order. Two diverged substantially, finding the exact minimum number of transfers; one also resolved cent ties by insertion rather than alphabetical order.
• In A pairs, separate tests crossed with separate code exposed deviation in one of three, below the required two. B implementations had deviations invisible to their own tests in all three.
• Inconclusive.

Why? First, separate tests checked properties, such as totals, ±1 cent and at most n−1 transfers, while explicitly leaving ambiguities open, so they could not fail on them. Second, interpretations were correlated: five of six implementations chose the same wrong reading of “few transfers”. Same model, same reading. Separation does not make interpretation errors independent. Third, tests written alongside code were coupled to it: one suite imported an internal constant and ran against no other implementation.

014: change the role rather than the model, reduced version

My idea: use agents with different characteristics rather than different models. Same specification as 013, the same six target implementations, two roles with three agents each:

• R, specification reviewer: list ambiguities and questions for the product owner, no code.
• Q, adversarial tester: choose the most plausible reading for each open point, declare it and enforce it with an exact test. Tests accepting either behaviour were forbidden.

The known deviations from 013 were D1, cent ties; D2, transfer strategy; D3, transfer order.

Reviewers: useful if at least two of three identified D1, D2 and D3.

Reviewer	Questions	D1	D2	D3	Proposed D2 answer
R1	11	✓	✓	✓	Equal pairs first, differing from intent
R2	12	✓	✓	✓	Equal pairs first, differing from intent
R3	13	✓	✓	✓	Equal pairs first, differing from intent

Three of three: rule satisfied. D1–D3 always appeared among the first four questions. But proposed answers repeated the implementers’ wrong readings. R3 also suggested insertion-order ties, A3’s deviation. The reviewer sees the ambiguity but would resolve it like the other agents.

Adversarial testers: better than neutral if at least two of three suites exposed deviations in at least two implementations. Baseline: 013’s neutral suites, at most one.

Suite	Implementations with exposed deviation	False alarms on correct code
Q1	2 of 6	16, noisy
Q2	6 of 6	8
Q3	6 of 6	4
Neutral 013 suites	1 of 6	1–2

Three of three: rule satisfied. Q2 and Q3 also found the mild deviation no 013 suite had detected, using a handwritten case with matching debt and credit. The price: every wrong reading by the tester becomes a red test against correct code. Q1 chose wrongly on two points, ties and error type, producing sixteen false alarms. In four of six implementations, its algorithm tests failed earlier for other reasons, hiding the real deviation. A red test means “these readings disagree”, not which is right.

Lesson

Separation alone is insufficient. With the same model and role, agents share interpretations, as 013 showed. Changing the role changes results in 014: the reviewer raises the right questions before coding; the adversarial tester makes them testable. Neither knows the right answer. Without a human decision, both repeat the common reading. Opposing agents turn silent ambiguity into an explicit question or a red test needing arbitration.

Conclusions supported by the data

These are small-sample conclusions, not general laws.

1. 100% coverage does not mean meaningful tests: the right test name with the wrong data.
2. Raw mutation scores mislead in both directions. The drop from 93.5% to 85.6% was noise: logging, equivalents and unrequested code. Postgres’s “100% detected” with slow fixtures was a timeout artefact. After verification, and with fast fixtures, SQLite and Postgres agree: 164 killed, the same 24 survivors, no real specification gaps.
3. With database and ORM code, the testing agent makes different mistakes: overly strict for the wrong reason, rather than too weak. Integration finds these; mutation testing does not.
4. The most exposed area is code nobody requested: a process problem.
5. Given feedback, the testing agent did not game the metric.
6. The strongest signal came from the pipeline rather than tests. CI that cannot fail makes test quality irrelevant, and no test metric would detect it. Read the logs.
7. AI must verify green too. The orchestrator reported an unverified result by trusting the status indicator.
8. SQLite hides exactly the bugs that matter: foreign keys, timezones, schemas and RLS. Its convenience concealed tests never running on the real database.
9. The author’s questions changed decisions. Asking why SQLite was used prevented a convenient workaround. Saying improvement was insufficient brought the file from 130 to 23 seconds, and CI’s Postgres suite from about nine to two minutes.
10. Precise specifications make separation unmeasurable: experiment 012 hit a ceiling.
11. With vague specifications, separation alone is insufficient: 013 showed the same model making the same wrong readings, while separate tests checked properties and left ambiguities unresolved.
12. Role matters more than separation: in 014, specification reviewers and adversarial testers exposed ambiguities behind the deviations, at the cost of false alarms and without knowing the correct answer. They generate questions for decision-makers rather than act as arbiters.

Declared limitations

This is what makes the experiment honest, and why I do not call it a demonstration.

• Two small features, with precise example-rich specifications written by an agent.
• The same model for testing and coding agents: both can misread the specification identically, making oracle and implementation agree on an error.
• Classifications and diagnoses are model judgements, from the orchestrator and a different-model reviewer, without human verification.
• Local Postgres was version 16; CI uses 15.
• The exact cause of mutant 11’s false kill in the slow run was not established. Fast fixtures correctly reported it as surviving.
• Experiments 012–014 each used one pure-logic feature, n = 3 per condition and one model, except Sonnet in 012 condition C. The orchestrator classified 014 reviewer questions.
• 014 was reduced: it reused 013 implementations known to the prompt-writing orchestrator, with no new neutral-condition replication.
• Possible contamination in 014: 013’s hidden intent was already public in the repository. Testers were instructed not to read it, and their recorded commands do not read repository files, but control was not cryptographic as in 012/013. Q3 almost matched the intent exactly: compatible with chance, but contamination cannot be ruled out.
• Models from different providers were not tested because session network configuration blocked them.

More intensive testing is needed: more features, replications, genuinely independent models and less polished specifications before declaring the first article’s argument confirmed or refuted. For now, there is a strong clue, that role matters more than separation, and an unexpected lesson: the real signal was in the pipeline.

Postscript. I soon encountered that unexpected lesson again in a completely different context: another agent, Codex, another tool and a legacy PHP-to-Laravel migration, once again with a gate saying “done” when it was not. If false green interests you more than mutation testing, read From legacy PHP to verifiable parity.

Practical tool notes

• mutmut 3.x: paths_to_mutate does not limit mutated files; it is a deprecated alias for source_paths. The first attempt generated mutants across the whole application, 763 MB. Use only_mutate.
• mutmut 3 does not mutate module-level constants. The orchestrator verified this with eight handwritten mutants, all killed, but the tool alone would not have seen them.
• mutmut 3 on a real database: timeout = (estimate + 1) × 15, with estimates apparently omitting fixture cost; tests ordered by estimated duration; multiple processes need one database each. Fast fixtures eliminate the problem.
• GitHub Actions without explicit shell: bash uses bash -e without pipefail. cmd | tee file hides cmd failure.

Quotations from the testing agent, rendered back into English

• “The specification only requires ValueError to be raised, not its text.”
• “With break, a malformed import row silently loses all subsequent events.”
• “Testing message wording would be brittle and would not protect anything users rely on.”

---

Sources in the repository

• PR MartoEporedia/taita#679, merged: features, tests, specs/010-school-days-mutation/ and specs/011-bridge-notifications-mutation/ with experiment.md.
• PR MartoEporedia/taita#680, merged: pipefail, tests migrated to Postgres, fast fixtures.
• PR MartoEporedia/taita#681, merged: specs/012-carpool-rotation/ and specs/013-expense-split/ with experiment.md.
• PR MartoEporedia/taita#682, open: specs/014-agent-roles/ with protocol, reviewer questions, suites and results.