Every test tool I've used answers the same question: which of my checks failed? It runs your assertions and tells you which ones went red. Useful — but it can only ever report on tests that exist. It is structurally blind to the test you never wrote. For LLM systems that blind spot is the expensive one. The regression that reaches a user is rarely "a check I had went red." It's usually "there was no check there at all." And a green suite looks identical in both cases. Here's the smallest concrete version I can show you. A suite that's green for the wrong reason A support-bot prompt with three rules: cite a source, never promise a refund, and reply in English. There's a promptfoo suite for it that asserts the first two — a contains for the citation, a not-contains for refund language. It passes. In CI it's a green check, and you'd move on. Now delete the "reply in English" line from the prompt and rerun the same suite. The bot's output changes — it can now answer in another language. And every assertion still passes, because nothing in the suite ever looked at the language: #1 [HIGH] SURVIVED [drop_instruction_lines] dropped line: "- Always reply in English." fix: add checks.llm_judge("the reply still follows: - Always reply in English.") The suite wasn't green because the system was fine. It was green because it wasn't looking. No individual assertion is wrong; the coverage has a hole exactly the shape of the rule nobody thought to test. Finding the hole instead of guessing at it You can't grep for a test that isn't there. So muteval comes at it from the other side: it degrades the system on purpose — deletes a rule, weakens a modal, drops a retrieved doc — reruns your existing suite against each degraded version, and watches for the case where the output changed but every check still passed. That case is a "survivor," and a survivor is the tool telling you: there is a behavior here your suite has no opinion about. That's the whole idea, and it's a different question than any assertion-runner asks. A normal suite tells you which of your tests failed. This tells you which test you're missing — the absence, not the failure. (It ran offline, no API key, in about a second — a deterministic mock model standing in for the bot, so the survivor is the suite's blind spot, not model noise.) What it is, and isn't A survivor is a candidate gap, not a verdict. Some are behaviors you intentionally don't test; you look and decide. muteval surfaces the hole, you rule on whether it matters. It's a per-suite diagnostic — "your suite doesn't cover this," not "here's a universal flaw in evaluation." It can only degrade the system in ways it knows how to (delete a line, corrupt a doc, …); a behavior no mutation exercises won't surface. It finds the holes near the edits it can make, not every possible hole. Those limits are real, and it's still the only thing I've found that answers "what am I not testing?" instead of "what did I test that broke?" The question Think about your own eval suite for a second: if someone quietly deleted one line from your system prompt tonight, is there a check that would go red before a user noticed? If you're not sure — that uncertainty is the gap, and it's exactly the thing that's invisible until you go looking for it. Repo (the offline demo above is in it, keyless): https://github.com/AshwinUgale/muteval