Hey everyone. This time I'll go through what got me started on this benchmark, and the core of how it's actually built. It started from reading complaints, not from an idea. The same ones kept coming up: numbers a vendor publishes don't match numbers someone else measures, swapping the model that does the grading moves the results more than the gap between the systems being compared, and because of that nobody really uses published numbers anyway. They test two options on their own data and keep whichever annoys them less. I work at Wontopos, and we sell a memory API, so I can't point at that and shrug. All numbers below are from the current build. Nothing is final yet, so some of them will have moved by the time you read this. Fairness first, because you have no reason to trust me I know how "I built a fair benchmark" sounds coming from someone at a memory company. So instead of promising anything, here's what's written in the repo. Wontopos hosts it and keeps it running. That is the whole role. We also build memory infrastructure, which means we compete in the thing we administer, so the limits are written down rather than promised: Our submissions go through the same approval as everyone else's. We do not merge our own. We do not decide who is admitted. The rules do. Our numbers are verified the same way as everyone else's. Wontopos publishes nothing on a new version for fourteen days. If other memory companies want to co-administer, that's better than us alone, and the offer is open. A few of the submission rules point the same way: Publish the per-question record. Anyone can recompute the number from it. The aggregate is a claim, the record is the evidence. The reader, the judge and the prompts are set by the version. A submitter does not choose them. The harness has to be one a customer could use. A number produced through a path only its author can reach is not a number anyone else can get. Everything there is Apache 2.0, so if we ever become the problem, the whole thing can be taken and run elsewhere without asking us. It's all here: github.com/wontopos/glasshouse. Fair warning about what you'll find. The benchmark itself isn't in there yet and submissions/ is empty. The rules went up first, and we haven't submitted either. The corpus One person's life, told across about 17 months of conversation, with the facts you're supposed to remember buried inside it. 103,572 turns, 1,991 sessions, roughly 1.9M tokens. It runs at four haystack sizes, from a small core up to the whole thing. The 1,882 turns that actually contain the answers are identical in all four, character for character. What changes is how much unrelated conversation is packed around them, so you can watch a system degrade as the haystack grows instead of getting one score and no idea what it means. The axes 1,547 questions across 14 axes at the full size. Most are ordinary recall: it was said, can you get it back, can you get it when the question uses different words, can you say when it happened, can you combine two facts. These four are why I built this. Stale facts. A value changed. The new one exists, but I made it hard to find on purpose. Three outcomes instead of two: Answer Score New value 1.0 "I don't know" 0.5 Old value, stated as current 0.0 The reason for the middle row is the whole point. Put "I don't know" and "confidently out of date" in the same bucket and you've hidden the thing that actually hurts in production, because those are very different things to be paying for. Contradictions. The conversation states two different values for the same fact, nobody corrects it, and nothing tells you which is right. There is no correct answer. Confidently picking one is wrong. Saying "these don't match" is right. Apparent contradictions. The mirror image. Two statements look like they clash but hold under different conditions, so both are true. Telling this apart from a real contradiction is the point. Abstaining. Questions about things never mentioned at all. An empty answer scores full marks, anything invented scores zero. On top of that there are 500 false-memory probes, which plant something that was never said inside the question itself and check whether the system plays along. Images 50 photos are shared inside the conversation, with 150 questions about them. Every one of those questions is pinned to a date: "In the photo from 14 April 2026, what was on the sofa?" That's there because with 50 photos, a question like "was the laptop open?" points at two of them at once. Inside the flow of the conversation that's fine, but a question lifted out on its own, or translated into another language, becomes impossible to answer correctly. The date narrows it to exactly one photo, and it doesn't leak anything, since nothing is asking when it happened. Languages The corpus carries 100 sessions in 10 languages, 10 sessions per language, mixed in with everything else. 800 questions run on this, in both directions: A fact stored only in another language, asked in English. A fact stored in English, asked in another language. Both directions matter because they break differently. One tests whether anything crosses the language boundary at all, the other tests whether the query side does. Speed, and why it's measured this way Speed matters, but measuring it fairly is harder than it looks, and this is the part I rewrote the most. The problem is geography. Raw latency includes the speed of light. Measure from Seoul against a server in the US and you're 190ms in before any work has happened. Report that as-is and you're ranking where the server sits, not how good it is. So the network floor gets measured separately and subtracted. What's left I call "latency minus round trip", not "pure compute", because response transfer doesn't fully subtract and calling it pure compute would be overstating it. Two conditions reduce the leftover error, and both are part of the procedure: Warm the connection first. TCP starts slow and ramps up. Without warming, a response crossing 14KB picks up an extra round trip, and a 3,700 token response sits right on that boundary. Record the response size. The remaining error scales with it, so writing it down lets a reader judge how much slack is in the number. The budget. People stop feeling like a conversation is flowing at around 1,000ms. The LLM's first token eats about 500ms of that on its own. So memory gets the remaining 500ms, and that's the target. Twice as fast as the target scores +1, four times slower scores -1, and it's a log scale in between, because the difference between 200ms and 400ms matters more than the difference between 3s and 3.2s. Per question, and capped. Each question scores between -1 and +1, and the results are averaged rather than summed. Sum penalties instead and the total sinks on its own as you add questions, which means the score stops describing the system at all. Speed is reported next to accuracy, not folded into it silently, and the weight is stated. And if speed can't be measured on a given system, the term is removed rather than zeroed. Being unmeasurable shouldn't be a penalty. Nothing has been scored yet, and I'm not quoting numbers before there are numbers. One question for you If you were going to run something like this, what would have to be in it before you believed the result? And if you've ever looked at a published memory benchmark score and thought "no", what tipped you off?