Your process is up. Cute. Is it correct?

I posted here 8ish months ago about how you should crash on unexpected panics. It was not safe for work. The post, and also the approach. This is the next iteration, a lot more nuanced, with the lessons that came from building an actual database, where in-memory state has to agree with itself or the whole thing is a very fast way to return wrong answers.

The premise has not changed. Everyone measures reliability by whether the process is still running, and that is the wrong metric. A reliable service is one whose invariants hold: the count matches the map, the index matches the data, the balance never goes negative. A panic lands between two writes, the default handler recovery swallows it, the process keeps serving, and now you have a service that is up, green on every dashboard, and quietly wrong forever. I wrote a ten-line cache that does exactly this. It reports two keys and stores one, and nothing in the stack thinks anything is off.

What changed is the nuance. "Always crash" is as dumb as "always recover". One malformed request restarting every instance is a denial of service waiting to happen. So the rule now is about boundaries: recover a panic only at the boundary that owns every side effect the panicking code could have made. An operation, a request, a durable job attempt, a worker, or the whole process. Recover at the narrowest one you can prove, and widen when you cannot. If no boundary inside the process qualifies, log the panic, exit, and let the supervisor start a clean process from durable state. Crashing is still the one recovery strategy that is always available, even when your recovery code is the thing that is broken. It is the last line of defense, not the first.

I recovered panics I should have let through. I shipped goroutines with no recover behind a supervisor that was misconfigured and slow to restart, and one goroutine panic took production down. I have lost stack traces to processes that died before anyone wrote them down. The post walks through the cache, fixes it twice, once by deleting the duplicated fact and once by exiting before the server can recover, and covers the supervisor side so a crash becomes a restart instead of an outage.

So, discussion flair. Where do you draw the boundary in your services? Do you wrap every long-running goroutine, or let the process die? Is anyone crashing on purpose in production, and what does the supervisor config look like? And if you think you know better and I am wrong, tell me.

Disclaimer: I wrote the article myself.