Five silent bottlenecks that turned a 400ms API into a 4.2s crawl No deploys. No schema changes. No new integrations. Just a scheduling and resource-planning SaaS with 40k users watching its p95 response time climb from 400ms to 4.2 seconds over six months. Nobody could point to a cause, because there wasn't one big cause, there were five small ones stacked on top of each other. This is the writeup of how we found and fixed them, without a rearchitecture. The symptoms before the numbers existed Support tickets called the dashboard "laggy" long before anyone had hard metrics. By the time it got measured: p95 latency: 400ms to 4.2s over ~6 months Trial-to-paid conversion down 11% in the same window Sales getting asked about performance during renewals The team had already tried the usual moves: bigger instances, a read replica, scheduled restarts. None of it held. When scaling vertically twice does nothing, that's a strong signal the problem is architectural, not capacity. Step one: measure before touching anything We spent the first week purely on instrumentation, tracing requests across the API gateway, app servers, database, and cache layer under normal load. No fixes yet. Changing the system while you're trying to measure it just adds noise. Five issues surfaced. None of them alone explained 4.2 seconds, but together they did. The five issues 1. N+1 queries in a hot path A dashboard endpoint fetched a customer's project list, then hit the DB once per project for status. 40 projects meant 41 round trips for one page load. 2. Connection pool exhaustion 8 app servers x 20 connections each = 160 possible connections, against a Postgres max_connections of 100. At peak, requests queued for a connection slot before a query even ran. 3. Cache invalidation nuking whole namespaces Redis had a 30s TTL on project status, reasonable on paper. But a background job flushed entire cache namespaces on any write, including unrelated ones. Actual hit ratio: 34%. 4. Cross-AZ chatter App tier and Redis cluster weren't pinned to the same AZ. ~40% of Redis calls crossed zones, adding 2 to 4ms each. At volume, that's hundreds of milliseconds per request. 5. A synchronous third-party call in the request path A usage-tracking webhook to an external analytics provider ran synchronously inside the request cycle. On a bad day that provider took 800ms to 1.5s, and every request waited on it. How we prioritized fixes Impact vs. risk, not ease of implementation: Sync third-party call, high impact, low risk N+1 queries, high impact, low risk Connection pool exhaustion, high impact, medium risk Cache invalidation, medium impact, medium risk Cross-AZ placement, medium impact, low risk We explicitly skipped touching instance sizes again. Two rounds of vertical scaling with zero improvement was itself the data point: the bottleneck was I/O and architecture, not compute. We also shipped fixes one at a time, measuring after each. Bundling everything into one release makes it impossible to know what actually helped, or what quietly broke something else. The fixes Move the analytics call off the request path // before: synchronous call blocking the response await analyticsClient.track(event); // 800ms-1.5s on slow days return response; // after: fire-and-forget via queue await queue.push('analytics.track', event); // ~2ms return response; Used the existing Redis infra as the queue backend, no new dependency. A worker process handles delivery with retries and a dead-letter queue. This alone cut 800ms to 1.5s off the median request on affected endpoints, and when the analytics provider had two outages during the engagement, users never noticed. Kill the N+1 -- before SELECT * FROM projects WHERE account_id = $1; -- then per project: SELECT * FROM project_status WHERE project_id = $1; -- after SELECT p.*, s.status, s.updated_at FROM projects p JOIN project_status s ON s.project_id = p.id WHERE p.account_id = $1; 41 round trips became 1. Query time for that endpoint went from ~620ms to ~45ms average. Retune the connection pool Dropped per-server pool size from 20 to 12 (8 x 12 = 96, under the 100 limit) and added PgBouncer in transaction pooling mode: [databases] app_db = host=127.0.0.1 port=5432 dbname=app_production [pgbouncer] pool_mode = transaction max_client_conn = 500 default_pool_size = 25 reserve_pool_size = 5 reserve_pool_timeout = 3 Connection wait time went from ~180ms average at peak to under 5ms. Fix cache invalidation Switched from namespace-wide flushes on any write to key-level invalidation tied to the specific project changed. Kept the 30s TTL as a safety net, but it stopped being the primary mechanism. Cache hit ratio: 34% to 91% within a week. Small diff, outsized effect on database load. Takeaway None of these fixes were exotic. This is what accumulated technical debt looks like in a system that grew for a few years without a dedicated performance pass. If your latency crept up gradually with no single obvious cause, look for a stack of small architectural issues before you reach for bigger hardware. Full writeup with more detail: Debugging latency in a high availability infrastructure setup Originally published on binadit.com