Why Data Locality Matters More Than Raw GPU Speed for RAG

The image below shows how each step of a RAG pipeline adds to the total response time, and which of those steps a faster GPU can actually help with.

A GPU upgrade can shorten query-embedding time if that step runs on the GPU. It can also speed up reranking, prompt processing, and text generation, if those happen on the GPU too. What it can’t do is remove the cost of reaching a database that lives somewhere else. A single request to a remote database can involve:

Looking up the address (DNS)

Opening the connection (TCP)

Setting up a secure channel (TLS)

The trip across the network itself

Routing between providers

Waiting in line at the database

Running the actual query

Packaging up the results

Sending that data back

Reusing connections can skip the repeated address lookups and connection setup. But it can’t shrink the distance itself. And it doesn’t help with network congestion, changing routes, or the extra overhead of crossing from one provider to another.

A simple RAG demo might make one search call and one model call. Real production apps are usually more involved. Picture one that:

1. Turns the user’s question into an embedding.
2. Searches the vector index.
3. Filters the results by tenant, date, permissions, or product.
4. Pulls the full document chunks.
5. Fetches metadata from a separate table.
6. Reranks the candidates.
7. Builds the final prompt.
8. Sends that prompt to the model.
9. Runs the model.
10. Goes back for more information if the first batch of evidence wasn’t enough.

Agent-style RAG workflows can repeat steps 1 through 10 several times. The model might search one source, look at what it got back, rewrite its question, call another tool, and gather more evidence before it finally answers.

Each one of these remote steps adds a little more waiting time. Once that waiting time adds up to more than the GPU’s share of the work, a faster GPU stops making much difference.

Location matters more than GPU speed, but you can’t just claim that. You have to measure it. Marketing numbers for GPU speed, or a generic internet speed test, won’t tell you anything useful. You need to measure real retrieval calls, timed from the machine actually running the model.

The right test moves the database’s location while keeping everything else (the database setup, the GPU, the data, the queries, the app itself) exactly the same. At minimum, a fair test should keep these things fixed:

DigitalOcean Managed PostgreSQL already comes with pgvector, which lets PostgreSQL store and search a few kinds of embeddings:

• vector - standard, full-size embeddings
• halfvec - half-precision embeddings that take up less space
• sparsevec - embeddings that are mostly zeros

It supports exact search as well as two faster, approximate search methods called HNSW and IVFFlat. DigitalOcean also supports pgvectorscale, an add-on that includes a technology called StreamingDiskANN for handling vector collections too large to fit in memory. You can read more in the DigitalOcean Managed PostgreSQL vector-search documentation.

To test location properly, keep the database engine, data, index, queries, and GPU setup the same, and change only where the database sits. That way, location is the only thing you’re actually measuring. You’ll want to compare at least three setups:

Same datacenter: The GPU Droplet and the PostgreSQL database sit in the same DigitalOcean datacenter.

Different region: The same GPU setup talks to an identical PostgreSQL database, but in a different region. Everything else (engine, plan, data, embeddings, index, queries) stays the same.

A competing platform: We used Modal here, calling out to a separate managed PostgreSQL database.

For the competing-platform comparison, you should also control where the compute and the database physically sit. Modal’s own documentation on region selection lets developers choose where their containers run and recommends placing latency-sensitive compute close to the database, since crossing regions adds real, measurable delay.

Picking a fast inference platform doesn’t solve the location problem by itself. You still need your database near your compute. Otherwise the delay from physical distance, network routing, and moving between providers can cancel out whatever speed gains you got elsewhere.

Run the tests in a random order

Randomize or interleave your three setups. Send one query to the same-datacenter setup, the next to the different-region setup, then the next to the competing platform, and keep cycling through.

This keeps timing from skewing your results. If you run the same-datacenter test in the morning and the different-region test hours later, changes in database load or network traffic could shift the numbers. You’d end up blaming location for a difference that was really just caused by testing at different times.

Report warm and cold connections separately

Apps that run continuously usually keep a pool of open connections and mostly use warm ones. Apps that scale down to zero, common in serverless setups, have to open new connections more often.

If you average warm and cold results together, you get a number that doesn’t accurately describe either case. Report them separately instead.

Think of the network penalty for one cross-region database call as a fixed cost. Call it N. A simple RAG pipeline that makes one remote database call pays N once. An agent-style pipeline that retrieves, checks metadata, and then searches again pays it once for each sequential call, because each call has to finish before the next one can start. If it makes k such calls in a row, the total penalty is N × k, not N.

A GPU upgrade, by contrast, speeds up one step, once, no matter how many times your pipeline calls the database. So the more sequential remote calls a pipeline makes, the more the network penalty multiplies, while the GPU’s contribution stays flat.

This is the shape of the argument, not a stand-in for your own numbers. Whether this actually matters for your app depends on your own N and your own k. The only way to find out is to run the test described above on your own pipeline, with your own regions and your own call pattern, and see what comes back.

A request to a database can’t travel faster than the signal carrying it can move across the physical distance between the two machines. Light moves a little slower through fiber-optic cable than through open air. And in a real network, data doesn’t travel in a straight line. Along the way, it typically passes through:

• Switches
• Routers
• Firewalls
• Gateways
• The boundary between cloud providers
• Sometimes crowded network segments

That sets a floor on delay that no amount of software tuning can get around.

Bandwidth and latency are two different things. Bandwidth is how much data can move per second. Latency is how long it takes for that data to get from one place to another. A connection with plenty of bandwidth can move a lot of data quickly once it starts flowing, but that doesn’t mean the first byte will arrive any sooner.

DigitalOcean GPU Droplets support up to 25 Gbps over private networking and 10 Gbps over public networking. Those numbers describe how much data can move, not how long it takes for a request to reach a database in another region and come back.

Vector-database requests tend to care more about latency than bandwidth. A query embedding is usually just a few kilobytes, and the results coming back aren’t large either. Moving that small amount of data doesn’t need much bandwidth. What actually slows things down is the round trip itself (the time spent waiting for the request to arrive and the answer to come back), plus time waiting in line at the database, plus the time the search itself takes.

Tail latency makes this worse. The median response time might look fine, while the slowest 5% or 1% of requests take much longer, because of network congestion, changing routes, database backlog, new connections being opened, or other workloads competing for the same resources. That directly affects how long a user waits for the first piece of the answer. The model can’t start building a response until the data it needs has actually arrived. Because of this, the time to the first useful piece of output is a more honest measure of real-world speed than GPU performance alone. It accounts for every step the user is actually waiting on, from fetching evidence and building the prompt to the GPU queue, prompt processing, and the first bit of output.

A GPU benchmark measures things like prompt-processing speed, time to first output, and generation speed. But users experience the whole pipeline, not just the GPU’s share of it. If the data lives far away, network and database delays can eat up, or even outweigh, whatever time you saved with a faster chip.

Two services can share a general location on a map but still run on separate clouds, separate datacenters, separate zones, or separate networks. Traffic can still cross a provider boundary, or travel over the open internet. The real goal is to put the app, the search layer, and the model in the same datacenter, connected privately if possible.

DigitalOcean lets you choose which datacenter a GPU Droplet runs in. A Managed PostgreSQL cluster running pgvector can sit in that same broader setup.

Doing this gives you a few things:

• Less distance for requests to travel.
• No unnecessary hops between cloud providers.
• Less exposure to outside network noise.
• Often simpler database access and network security.
• Potentially lower data-transfer costs.
• An easier system to monitor end to end.

To be clear, this doesn’t make the database behave like local GPU memory. Managed PostgreSQL is still a separate service running somewhere else. The point is just to put that unavoidable remote step as close as the platform allows.

Teams should confirm this by measuring retrieval time directly from the machine running the model. Looking only at how long the database itself took to run a query isn’t enough. It misses the time spent getting to and from PostgreSQL.

The chart below compares the time your app actually measures against the time PostgreSQL reports for itself. The gap between the two shows how much delay is being added outside the database, by things like opening the connection, the network round trip, waiting in line, packaging results, sending them back, and handling them in your app.

Tuning the HNSW index might shave off some of the 6 ms the database itself takes. But that tuning won’t recover most of the other 69 ms. Look at where things are physically located, how traffic is routed, how connections are pooled, how results are packaged and sized, and how your app coordinates all of this, before assuming the database itself is the problem.

A faster GPU lets you train bigger models, use more context, run bigger batches, and get more done per second, including faster prompt processing and faster generation. But it’s a mistake to assume a GPU upgrade will improve your whole pipeline by the same amount, because the GPU only touches part of what happens during a request. Retrieval, metadata lookups, and network transfer all happen before the GPU ever sees the prompt, and none of those get faster just because the GPU did.

Here’s the shape of the comparison, without pretending to know your numbers. A RAG pipeline’s total response time is split across several steps: retrieval, reranking, prompt building, prompt processing on the GPU, and generation, plus some network overhead in between. A GPU upgrade only speeds up the steps that actually run on the GPU, mainly prompt processing and generation. Every other step, including however long retrieval takes, stays exactly the same no matter how fast the new GPU is.

If retrieval is a small share of your total time, a GPU upgrade will show up clearly in your numbers. If retrieval is a large share, because your database sits far from your model, a GPU upgrade can speed up the steps it touches and still leave your total response time barely changed, since the steps it can’t touch are still eating most of the clock.

The only way to know which case you’re in is to break your own pipeline down step by step and measure where the time actually goes before deciding what to fix.

Use this table to decide when to focus on data location, and when something else, like generation speed, cost, retrieval quality, or reliability, deserves your attention instead.

For a RAG app on DigitalOcean, a sensible starting point is to put GPU Droplets and Managed PostgreSQL with pgvector in the same region. Use persistent database connections, private networking where you can, and measure end-to-end time from the machine running the model. From there, compare four possible fixes:

1. Tune the vector index and the SQL query.
2. Cut down on repeated retrieval and metadata calls.
3. Move retrieval and the model into the same region.
4. Upgrade the GPU.

Put your effort into whichever step is actually costing the most time, based on what you measure.

Other platforms face the same physics. Modal lets developers pin their compute to a region and recommends running latency-sensitive work close to the database. That’s a real way to help with location, but it also proves the point: a fast inference platform doesn’t automatically solve where your data lives.

RAG response time depends on the whole path a request takes, not just the GPU. Teams that compare GPUs without looking at where their database sits are often optimizing the part they can see, not the part that’s actually slowing them down. A faster GPU speeds up GPU work. Putting your data closer removes the wasted time that happens before the GPU even gets to work. In systems that make repeated searches, retrievals, metadata lookups, reranking calls, and tool calls, those small savings add up fast.

You’ll see this show up in how fast users see a first response, in your slowest requests, in how many requests you can handle at once, in your infrastructure bill, and in how well you’re actually using your GPU. None of this means location always matters more than compute. It means you have to measure the whole path before deciding a faster GPU is the fix.

If the time your app measures for retrieval is a lot higher than what the database reports for itself, that’s your answer. Move the data closer, cut unnecessary remote calls, and stabilize retrieval before you spend money on a new GPU.

For most production RAG systems, the fastest way to a faster answer isn’t a faster GPU. It’s a shorter distance to your data.

• Managed PostgreSQL for Vector Search: overview of PostgreSQL vector search, filtering, and pgvector indexes.
• Best Practices for PostgreSQL Vector Search: guidance on exact search, HNSW, IVFFlat, filtering, reranking, and query tuning.
• Advanced Vector Workloads with pgvectorscale and Hybrid Search: docs on StreamingDiskANN and Statistical Binary Quantization for large, disk-based vector search.
• Managed PostgreSQL Vector How-Tos: how to turn on pgvector on a managed PostgreSQL cluster.
• pgvector: Open-source vector similarity search for Postgres: the project’s own docs on vector, halfvec, and sparsevec types, search methods, and index setup.
• Regional Availability: which DigitalOcean products are available in which datacenters.
• Droplet Features: GPU Droplet specs, including network bandwidth limits.
• Virtual Private Cloud Documentation: docs on private networking between DigitalOcean resources.
• Modal: Region Selection: docs on pinning compute to a region to reduce cross-region latency.