Backend
The request we could not follow across four services
Sergey Shinder DEV Community
2 views
A customer sent us a screenshot of a 500 error with a timestamp. That was all we had. The request had passed through an API gateway, an auth service, an orders service, and a payments service, and each of them logged beautifully in its own format, into its own index, with no shared identifier anywhere. Finding that one request meant grepping four log stores by timestamp window and guessing which lines belonged together.
We got it eventually, about ninety minutes later, by correlating on a user ID that happened to appear in three of the four services. The fourth logged only an internal account key, so we joined through a database query. Ninety minutes to answer "what happened to this one request." That's not an observability problem you fix with more logs. We had plenty of logs. We had no way to connect them.
The missing piece was a correlation ID: one identifier generated at the edge, passed through every hop, and written into every log line. It sounds almost too simple to matter until you've had it, and then you can't work without it. One field, and the ninety-minute archaeology becomes a single query returning the full path of a request in order.
Retrofitting it was less about technology than about boundaries. Every internal HTTP call had to propagate the header. Every async message needed the ID carried in its envelope, which is where most of the work was, because queues are exactly where causal chains normally get severed. Every log statement needed the ID in structured context rather than interpolated into a message string, so it could be indexed instead of regexed. We used the W3C trace context header so the same ID also fed our tracing backend for free.
What surprised me was how much the shape of the system changed once we could see it. Requests we thought made two downstream calls made nine. A retry loop we didn't know about doubled traffic to one service. A cache we assumed was hot was being missed constantly, visible instantly in the trace as a gap.
Logs describe events. Traces describe causality. If you can't follow one request end to end, you're not debugging a distributed system, you're interviewing witnesses who never met.
– Sergey Shinder
Read original: https://dev.to/sergey_shinder_ab2d943365/the-request-we-could-not-follow-across-four-services-1p0o
← Previous
I built a salon website that its owner can run without a developer
Next →
ArgoCD SSO with AWS IAM Identity Center (via Dex SAML)
Related
I built a message board for AI agents. Four bugs shipped past a 100% green test suite.
I built agora, a public message board that any AI agent can read and post to. No account, no signup, no API key — only body is required. Agents leave notes, "here's what broke and here's what fixed it" field notes, or questions other agents can reply to. It's about 1,800 lines of PHP with a SQLite
DEV Community
Principles of Memory Management in Java
submitted by /u/davidalayachew [link] [留言]
Reddit r/programming
The Best Tools for Automating Marketing Graphics in 2026
Bannerbear, Picnie, Placid, Creatomate, BannerBoo, and what to consider before choosing a creative automation platform. Marketing teams rarely have a problem coming up with new campaigns. The problem usually begins after a campaign is approved. One design may need to become a dozen social posts. A
Dev.to (EN Zone)