Last week I published a post about nine measurement bugs I found in my own evaluation harness, all of which happened to favour my results. Two readers replied with things I hadn't found. Both were about my checks rather than my numbers. I verified both and fixed both. My results did not change at all. That turned out to be the interesting part. Reader one: a canary that could not detect what it claimed to Some setup. The harness mutates a source file in a temp copy and runs the test suite against it. To prove that mutations actually reach the interpreter, I had a canary: overwrite the module with unparseable garbage and assert the suite fails. Vinh Nguyen pointed out that CPython's default bytecode invalidation keys on source mtime in whole seconds plus source size. A mutation flipping < to > is byte-size identical. If a stale .pyc is sitting there with a matching header, Python skips recompilation and imports the original bytecode. The mutation is on disk. The original behaviour executes. The suite passes. The mutant gets recorded as surviving. Here is why my canary could not catch that. Unparseable garbage has a different byte length. So the pyc genuinely does invalidate, so the canary fires, so it reports success. It was answering a question I wasn't asking. I checked the mechanism first. # compile a pyc from this a == b # overwrite with a byte-identical mutation, force mtime back a != b # import: stale bytecode executes, original comparison runs Reproduced on the first try. Then I checked whether it reaches anything in my harness. It doesn't. Every temp-copy call site passes ignore_patterns("__pycache__", "*.pyc", ...), so no pyc ever arrives in the copy. I verified it by copying a target with 19 real pyc files and confirming zero arrive. Here is the part I keep thinking about. I wrote that line to avoid copying junk. It turned out to be load-bearing for correctness, and I had no idea until someone pointed at the mechanism. The reason an instrument is correct is often not the reason you wrote the code that makes it correct. A line added for tidiness that silently does safety work is a line someone removes during a cleanup, with no way to know what it was holding up. I added a byte-size-preserving canary alongside the original. Eight of twelve targets are eligible and all pass. The other four report N/A rather than being silently skipped. Reader two: the negative control I didn't have Ahmet Özel came at it from a different direction. His reading of the original post was that "debugging is triggered by surprise" explains a one-directional bias without anyone needing to be dishonest, which is exactly the argument I was making. His counter-measure was one I didn't have. Keep a deliberate negative control. A case where the score should be near zero. Everywhere else in a harness, high is good and low prompts investigation. On a negative control, high is the alarm. That gives you one place where a flattering failure is the surprising one, which is the only condition under which debugging reliably fires. He also made a sharper point about pre-registration, which I was doing. Writing down your expected number helps when you're wrong. It does nothing when the instrument is broken in a way that produces the number you predicted. One of my original nine was exactly that. A classifier misread self.assertEqual as "no assertion" while I was testing a hypothesis that models write assertion-free tests. It would have handed me my own prediction with a confirmation attached. So I built the control. A suite of deliberately vacuous tests. Import the module, call things, assert nothing meaningful. I expected near zero. It killed 7 of 51 mutants. 13.7%. That is not a bug. assert x is not None is a real detector, just a very narrow one. It catches exactly one class of mutation: a function that started returning None. All seven kills were that operator. Zero were constant mutations and zero were comparison mutations. So a suite of vacuous tests has a non-zero floor rather than a zero one. That is a calibration fact about my scoring that I would never have learned from a control that came back clean. The design constraint is the practical part, and it took me a second attempt to get right. A negative control has to fail loudly if it passes for a boring reason. A near-zero score is also what you get when the module never loaded, which is precisely what bug 1 in my last post produced. Three targets at 0.000 that read as a finding rather than as a fault. So the control asserts separately that the suite passes on clean source, that the module actually imported and executed (coverage-confirmed, 37 lines), and that both canaries still fire. Round two: my fix asserted a proxy Vinh came back after I described the fix and found a hole in it. My regression test asserted pyc_count == 0 in the temp copy. But set PYTHONPYCACHEPREFIX and bytecode goes to a central tree keyed on the copy's absolute path. Zero pyc files arrive in the copy and the stale read happens anyway. My test would have passed while the harness was lying. He ran three branches on his own machine to isolate the real dependency: no prefix, fixed work path: fresh prefix, same fixed work path: stale, zero pyc in the copy prefix, unique directory per mutant: fresh I reproduced all three. What actually protects me is a second accidental thing. Every copy site uses tempfile.TemporaryDirectory(), which gives a unique path per mutant, which is branch three. I verified it across every call site and ran a full twelve-target sweep with the prefix set. It diffs byte-identical against the committed results. Two accidental protections. Neither written for the reason it works. ignore_patterns to avoid junk. TemporaryDirectory because it cleans itself up. Both are now commented as load-bearing and name the environment variable explicitly, because anyone adding a --work-dir flag would remove one of them and have no way to know. The rule underneath this is the most useful thing I got out of the whole exchange. My test asserted a proxy. Pyc count is a proxy. "The observed behaviour actually changed" is the property. The proxy held under my configuration and failed under a supported environment variable. The property costs the same to assert and does not have that failure mode. So I rewrote the test to push a byte-size-preserving mutant through the real code path and assert the observed outcome isn't "survived." I added a meta-test that reproduces the stale branch to prove the check has teeth. A regression test nobody has ever seen fail sits in the same category as the canary that couldn't detect this in the first place. One result that came along the way While I was in there I closed something I had deferred: hand-labelling the nine mutants the agent couldn't kill. Seven are provably equivalent, six of them type-hint mutations inside TYPE_CHECKING blocks that are never evaluated at runtime. Two are genuine misses, traced to the agent testing the wrong call pattern and the wrong function. So the claim moves from 44 of 53 with an unbounded equivalence caveat to 44 of 46 killable, 95.7%, with the per-mutant arguments published. What I actually took from this Two rounds. Two readers. Four checks improved. Zero numbers moved. Nobody disputed a finding. All of the scrutiny landed on the instrument, and both times the instrument was wrong in a way I could not have seen by reading it. That is the argument for publishing the harness rather than the headline. A result is a claim people can take or leave. An instrument is something they can attack, and the attacks are what tell you whether it works. I got more out of two comment threads than out of forty hours of my own review. Two things worth stealing: Assert the property, not the proxy. Same cost. One fewer failure mode. Build one check where a good number is the alarm. Everywhere else, surprise is what triggers debugging, and a flattering bug never surprises you.