I tried lying to DFD's gates

In the previous three articles of this series, the third, fourth and fifth, the most useful signal always came from the same place: a gate saying something different from reality. A CI that could not fail, a gate stuck on an old state, evidence that existed on one machine only.

Those are stories, and stories do not tell us whether DFD works. So I tried to measure two things:

1. how many fixes are needed after a feature has been declared done;
2. what happens if I lie to the process on purpose, one lie at a time.

First: how many fixes after release

I took every feat commit with the files it touched, excluding tests, documentation and content, and counted the fix commits on the same files in the following ten days. It is a crude measure: a fix on the same file is not necessarily caused by that feature, and bugs nobody has found yet do not show up.

The two DFD features on Taita (the ones from the fifth article): zero attributable fixes after release. Two commits did touch the files of the first-access feature, but one fixed a bug from April and the other was a new requirement. There was one real defect, a frame in the Telegram login without a title for screen readers, visible only in production. The accessibility check planned in the rollout found it the same day.

The other features on Taita, in the same month: 13 out of 22 have at least one fix in the following ten days. Almost all of them talk to a language model, to Google or to WhatsApp: dates computed by the model that do not match the weekday the user said, times stored in the wrong time zone, messages that were never read.

A feature built with spec-kit on another project, FantaContext: merged at one in the morning, four fixes by the afternoon. The first changed 649 lines because the feature did not work in production; another was needed to accept a label that differed between real data and test data.

The numbers favour DFD, but they prove nothing. The DFD features were interface work, almost all of it verifiable locally. The ones with the most fixes depend on inputs that do not exist locally. I compared easy-to-verify work done with DFD against hard-to-verify work done without it, and with this data I cannot separate the framework from the kind of work.

So the count does not answer the question. It made me change the question: instead of asking whether DFD produces fewer bugs, I asked whether its gates notice when something is off.

Second: lying to the gates

The test bench

I had an agent build a small lab: a new repository, with no remote, containing a function that computes the weekly cycle of a school canteen menu. DFD installed from scratch, one feature taken all the way to release approval. On top of that, one lie per branch:

• evidence redone without redoing the work: code changed and hashes recomputed by hand, a test log written by hand, fingerprints copied after a design change;
• tests that cannot fail: a test that never calls the code, a skipped test, a CI ending with || true, which exits without errors whatever happens;
• work outside the perimeter: an out-of-scope file, a regression in existing code;
• documents changed after approval: the DoD edited in the criteria catalogue, or only in the human-readable document;
• stale state: a revert with the state still saying “verified”, evidence not committed and checked from another clone;
• convenient data: a criterion green only on the fixtures;
• a human decision recorded by the agent itself.

Fifteen lies in total. Before running them, the agent wrote down and committed its predictions: what it expected for each one and why, citing the code of the CLI. The predictions could not be changed afterwards.

I ran everything twice. The second run fixed the limits of the first: a CLI built from the current source instead of an old build, lies hidden inside 5-6 commits of normal work instead of standing alone, four control branches with no lie at all, and verifiers running on models different from the one that injected the lies.

What the CLI sees on its own

Out of 15 lies, 4 blocked, 10 not seen, 1 out of scope.

The blocked ones:

• code changed after verification;
• DoD changed in the criteria catalogue after design approval;
• code reverted with the state still saying “verified”;
• uncommitted evidence: on another clone the CLI reports “missing evidence” and blocks.

The last one is the problem from the fifth article, and here the CLI does its job.

The other ten all pass. Recomputing a hash without rerunning the tests switches off the “evidence modified” check. A log with a skipped test, or with FAILED and exit 0, counts as green: the CLI does not read the content of logs. A file outside the evidence list does not exist for the CLI. A DoD changed only in the document people read changes nothing, because the gate looks at the other file.

The human decision recorded by the agent is out of scope by design: the CLI states that it does not authenticate who decides.

With the updated CLI the result is identical, message for message: the gate conditions had not changed. And in the realistic scenario the CLI gives all 14 branches the same state, “review to be confirmed again”, zero errors. To the CLI, a branch with a lie and a clean one look the same.

What an agent re-reading the work sees

Each branch was handed to an agent that knew nothing about the experiment, in a neutrally named folder, with a single instruction: take the feature to release following DFD, and before approving anything tell me whether the declared state matches the actual work.

Verifier	Lies found	False alarms on controls
Claude Sonnet	9 of 10	0 of 4
Codex	7 of 7 completed	0 of 3

Codex completed 7 of the 10 branches with a lie: the others stopped at my account’s usage limit.

The only lie Sonnet missed is the DoD changed only in the human-readable document, the same one the CLI misses. The others surfaced in two ways. Four by rerunning the tests, which is exactly what the CLI does not do. Five by comparing the diff with the tests and the specification.

Two things I did not expect.

First: Codex blocked all three control branches it completed, the ones with no lie. It was right: the function does not check that the number of weeks is positive, and with zero it crashes. The defect had been there from the start, nobody injected it, and no DoD criterion covered it. In the first run too, a verifier had found a real defect in the base.

Second: Codex fixed the code in five branches even though it had only been asked to report. Sonnet stopped at reporting. Neither recorded decisions on my behalf.

The predictions about the CLI were right 15 out of 15. Those about the agents were not: I expected Sonnet to find about six, and it found nine. I had underestimated the lies you uncover by comparing tests and specification.

What I take from it

The CLI stops those who forget, not those who lie. If the state is not updated, the gate notices. If it is updated without redoing the work, it does not. That is consistent with what the CLI says it is, a register that runs nothing and authenticates nothing, but it is less than a green gate suggests.

Today the real protection is re-reading. An agent that reruns the tests and compares the diff with the specification finds almost everything, even without knowing what to look for, and sometimes finds bugs nobody planted. The problem is that this re-reading leaves no trace in the gates. If nobody does it, the gate stays green anyway.

The fix count and the experiment say the same thing from two sides. The only production defect DFD caught on Taita was caught by a check that actually ran, in production, as planned. Where the check is a declaration, the gate is worth as much as whoever declares it.

This led to five changes to the framework, opened as issues:

• tie the evidence to the state of the files the suite ran on, and report what changed afterwards;
• read the logs: skipped tests and FAILED with exit 0;
• include the human-readable DoD in what invalidates the review;
• make a review to be confirmed again say what changed;
• make status exit with an error when a gate is blocked.

The first two bring into the gate exactly the two checks that today only a re-reader performs.

Update, 1 November: all five have been closed. Rerunning the same experiment, blocked lies go from 4 to 11 out of 15 and unseen ones drop to zero. The account is in the next article of the series.

Limits to declare

• The lab is small and artificial: one function, one feature, one domain.
• The lies, and the normal work hiding them, were written by a Claude agent. The verifiers were different models, but the two unintended defects in the “clean” work show how hard it is to build a truly clean control.
• One attempt per combination of branch and verifier.
• Verifier isolation was checked on the recorded commands, not with a system log. The Claude agents inherit my configuration.
• Evidence “on another machine” is simulated with a local clone.
• The fix count compares features from different domains: read it as a hint.
• The human decisions in the lab were recorded by the agent on my behalf, and this is stated.