How to Benchmark LLM Inference Yourself: A Design Standard for Numbers You Can Trust

• Fifteen levers move a benchmark’s headline number, and they group into four families: warmth, load, workload, and measurement. Most appear accidentally. A reader who knows the list reads every benchmark differently.
• A concurrency-1 median did not predict load. On the live DigitalOcean run, median TTFT rose 18 percent from concurrency 1 to 50 (202.1 ms to 238.4 ms in window 1). TTFT p95 rose 2.8x (445.4 ms to 1,245.1 ms). Total completion p50 rose 3.0x (741.4 ms to 2,241.2 ms).
• Two honest reruns of the same live test still moved p99. Two sequential concurrency-1 windows, 300 requests each, differed 26 percent at TTFT p99 (822.3 ms vs 634.4 ms) and 5 percent at TTFT p50. A p99 “improvement” smaller than your rerun spread is not a finding.
• The client is part of the instrument. At concurrency 50, the OS-thread client burned 1.02 to 1.24 CPU-seconds per wall-second. The asyncio client stayed at 0.24 to 0.28. TTFT p50 agreed within 3 ms. TTFT p99 did not. A benchmark that does not disclose client architecture and generator utilization has an undisclosed instrument error.
• The deliverable is a one-page disclosure standard, versioned and linkable, with two stated uses: run your own benchmarks against it, and read anyone else’s with it as the interrogation list.

When to use this standard, and when not to

Use it when you are about to publish a latency or throughput number, when you are reading a vendor or aggregator benchmark, or when you need to decide whether a reported p99 delta is larger than the noise floor.

Do not use a concurrency-1 median as production evidence. Do not read the live tables below as “DigitalOcean Serverless Inference latency is X.” They are protocol deltas on one model during one morning. Do not treat a c=50 cell with client CPU/wall above 0.7 as a pure server measurement.

Each row is a term you will see in the tables below, plus one picture of what it looks like in practice. The example numbers match this piece’s own runs.

Each lever below states the mechanism, the direction it biases, and whether it typically appears deliberately or accidentally. The families matter more than the count: once you see the four categories, you generate the interrogation questions yourself.

Four families, fifteen levers. Every lever moves the headline in a knowable direction, which is what makes each one a gaming vector, deliberate or accidental.

Let’s break down each of these levers in detail.

Warmth levers

1. Warm-only measurement. The first requests are slow while the model loads. Later requests are fast. If you throw away the slow ones and publish only the fast ones, you are showing the system’s best case as if it were every case. The number always looks better. This usually happens by accident: you sent a few test calls while debugging, then ran the benchmark on an already-warm system. The serverless consistency study shows this gap can be about 10x depending on when you measure.

2. Unreported cache state. If you send the same prompt twice, the engine can reuse work from the first call. You are then timing the cache, not the model. The prompt caching study measured this on one 3,110-token prompt: 462 ms the first time, 39 ms the second time. That is a 92 percent drop, and a reader never sees it if the write-up never mentions the cache. vLLM turns this reuse on by default. See vLLM’s engine docs. So if your benchmark repeats the same prompt on a default vLLM build, you are running a cache test unless you turned caching off or changed the start of each prompt. More on this in How to Use Prompt Caching and Prompt Caching in Practice.

3. First-hit exclusion. The very first request on a fresh GPU is extra slow. The engine is compiling code and setting up memory. That slowness is startup cost, not how the system runs once it is up. Dropping that first request is fine if you say you dropped it. It is misleading if you stay quiet, because the reader cannot tell whether “200 ms” includes startup. The Ornith 9B fine-tuning benchmark hit this slow first request and wrote it down. That is the honest version.

Load levers

4. One user presented as production. A test with one request at a time gives the GPU to a single request. Nobody else is waiting, so the number looks fast. Real traffic has many requests sharing the same GPU. They queue, they batch, they get in each other’s way. None of that shows up at concurrency 1. Raising concurrency does not stretch the same number. It changes what you are measuring. See When your vLLM p99 is worse than your p50 and Debugging p99 TTFT.

5. Throughput sold as latency. “This GPU does 2,000 tokens per second” is a batch number. It does not tell you how long your request took. Pack more requests into one batch and total tokens per second often go up while each user waits longer. Mixing the two always makes the vendor look better.

6. Cherry-picked concurrency. Model A wins at 5 users. Model B wins at 50. If you only publish the 5-user test, nobody can see the crossing. The fix is not a promise to be fair. The fix is a sweep: at least three load levels, side by side.

7. Unreported duration. A 60-second test misses problems that show up after a few minutes: queues that grow, GPUs that heat up, replicas that slow down. If the system falls over at minute five, a one-minute benchmark reports the easy part and skips the failure.

Workload levers

8. Favorable lengths. Short prompts make first-token time look good if the model is weak at reading long input. Long answers make total time look good if the model is strong at writing. You pick the length, you pick the winner. An Artificial Analysis study moved its default input to 10,000 tokens for this reason: the old short default did not match real use.

9. All requests are the same length. Production traffic is a mix: some short, some long. Long prompts steal GPU time from short ones that are already streaming. If every test request is the same size, you never see that pile-up. You are benchmarking a workload that does not exist. The serving-side view is stated in Continuous vs Static Batching tutorial.

10. Unreported sampling settings. Temperature, top-p, and max tokens change how long the answer is. Longer answers take more time. Two tests of the same model with different hidden settings are not the same test, and nobody can rerun them. More on this in How to Send Prompts to a Model Using the Chat Completions API.

11. The same name, a different model. FP8 and BF16 are different precision. One is smaller and often faster. Comparing them under one model name is a mismatch. Labeling both is a fair test. The consistency study found providers serving the same model name at different, unstated precisions. This happens in the wild and in production.

Measurement levers

12. Averages instead of percentiles. A mean blends the fast majority with the slow few and lands on a number almost nobody experienced. On this piece’s live run, concurrency-1 median TTFT was 202.1 ms while p99 was 822.3 ms. Publishing only the median hides the tail. You can read more about this in p50 vs p99 Latency and Debugging p99 TTFT.

13. Best-of-N run selection. You run the test five times. You publish the best one. That is not a typical result. It is the luckiest result. The common accident: you reran until the numbers “looked stable” and kept the last run.

14. Client-side bottlenecks. The laptop or Droplet sending the requests is part of the clock. If that machine is busy, it adds delay and you blame the server. A 2026 paper by Chandrasekar and Kramberger shows a single-process async tester can queue work on itself and inflate TTFT. Source: Identifying and Mitigating Systemic Measurement Bias in Production LLM Inference Benchmarks. If a write-up never says how the client ran, or how busy it was, you cannot tell server delay from tester delay. The live run later in this article that I ran measures that on DigitalOcean Serverless Inference.

15. Undisclosed network position. If you measure from a machine sitting next to the server, you skip the trip your users take. Light in fiber already needs 41.3 ms round trip between New York and San Francisco, and 153.3 ms between New York and Singapore, before the model does any work. Source: The Data Locality Tax. Raw records: anishsingh20/data-locality-tax. A latency number with no location is missing a term the size of the effect.

This list is about speed tests. Quality tests have their own tricks, including different harnesses scoring the same model differently. The Choosing the Right Model for Your Inference Use Case: Inference in Production Series covers those.

This section is a live API run, not a simulation. It extends the published p50 vs p99 Latency test and ran as one script on August 24, 2026 against https://inference.do-ai.run/v1. Raw records: anishsingh20/inference-benchmark-design-standard. Related measured evidence on the same endpoint family: anishsingh20/serverless-inference-tail-latency-study.

What I ran

• Target: https://inference.do-ai.run/v1, model ID mistral-3-14B from GET /v1/models, streaming Chat Completions.
• Client: GPU Droplet gpu-h200x1-141gb, 24 vCPUs, NYC2. The GPU was not the serving engine. It was the measurement machine, same role as in the p50 study.
• Trace: seed 7, 70/20/10 short/medium/long (200/100, 1000/300, 6000/600), per-request nonce prefix so prefix caching cannot swallow the run.
• Sample: 310 requests submitted, first 10 trace IDs discarded, 300 measured per cell. Ten cells, 3,000 measured requests, zero errors.
• Sampling: temperature 0, stream on, stream_options.include_usage on. Prompt tokens on the measured short class sat at 220 to 222. Long-class prompts reached 6,022 tokens. Completions filled the cap (100, 300, or 600).

Run A, the load lever. Identical seeded trace at concurrency 1, 8, and 50, thread arm.

Four readings, in the order a reader should take them.

1. Median TTFT is a weak load detector. Window 1 moved from 202.1 ms at c=1 to 238.4 ms at c=50, an 18 percent rise. Window 2 moved from 192.0 ms to 234.8 ms, a 22 percent rise. A concurrency-1 TTFT median would have told you almost nothing about c=50.
2. p95 TTFT is where load shows up. Window 1: 445.4 ms to 1,245.1 ms, 2.8x. Window 2: 363.7 ms to 1,174.7 ms, 3.2x. The p50 vs p99 study argued that tails, not medians, predict production. This sweep is that argument as a table.
3. Total completion time compounds. Window 1 total p50 rose from 741.4 ms to 2,241.2 ms, 3.0x. Window 2 rose from 748.2 ms to 2,176.4 ms, 2.9x. If your workload is an agent step that waits for the full answer, the concurrency-1 total median understates the c=50 median by about 3x on this run.
4. The c=50 thread client is part of the measurement. CPU/wall was 1.024 and 1.186. Above about 0.7, the generator is spending most of a core on itself. Read the c=50 tails as mixed server-plus-client numbers, not as a pure serving claim.

TTFT p99 was not monotonic meaning, it did not consistently move in one direction as concurrency increased. In this context, it means that the p99 value only increased or only decreased as you raised the concurrency level, without reversing direction. However, the observed sequence in Window 1 was 822.3 ms → 631.5 ms → 1,305.6 ms. So, if you only compared c=1 and c=8, you might wrongly conclude that higher concurrency improved p99 latency, but with the third point at c=50, the trend reverses.

Additionally, two runs at c=1 (sequential windows) still differed by 26 percent at TTFT p99 (822.3 ms vs 634.4 ms) and 5 percent at TTFT p50 (202.1 ms vs 192.0 ms). If a reported p99 change is smaller than the variation seen just by repeating the test, it is not a valid finding. The measured data here shows exactly that variation.

Run A. Median TTFT stays near 200 ms. p95 is the line that bends at c=50. Source: 300 measured requests per cell, mistral-3-14B, DigitalOcean Serverless Inference, August 24, 2026. Raw JSON in the evidence repo.

How to read this chart. There are two panels because we ran the same test twice, twelve minutes apart. Left is the first window. Right is the repeat. Across the bottom is how many requests were in flight at once: one at a time, then 8 overlapping, then 50 overlapping. Up the side is how long you waited for the first word of the reply, in milliseconds.

The blue line is the typical wait: half of requests were faster than this. It stays near 200 ms even at 50 overlapping requests. If you only published blue, you would say load barely mattered. The orange line is the unlucky 5 percent. It stays quiet at 1 and 8, then jumps to about 1,200 ms at 50. That is where load actually shows up. The red line is the unlucky 1 percent. It does not rise in a straight line. On the left it even drops from 822 ms to 632 ms before shooting to 1,306 ms. On the right it starts at 634 ms and ends at 1,500 ms. Same prompts, same machine, two honest reruns, and the slowest 1 percent already disagrees.

Run B, the client lever. Same trace, same concurrency 50, driven once by a single-process asyncio client and once by an OS-thread client. Request building, parsing, and timestamping are shared code. The only difference between arms is the concurrency mechanism.

• Asyncio client: This is a Python client built using the asyncio library, which manages concurrency by running many tasks on a single thread using asynchronous, non-blocking I/O. Each request uses a coroutine, and the event loop schedules many requests without the overhead of operating system (OS) threads. This is efficient, particularly when the network or server is the primary source of delay, and the client is not CPU-bound.

• OS-thread client (sometimes referred to as “threaded” client): This client uses the Python threading library (or similar), where each concurrent request is managed by a separate operating system thread. This approach is more parallel (each thread can run on a different CPU core) and can be helpful if the client-side code is CPU-bound. However, it incurs higher overhead, especially in Python due to the Global Interpreter Lock (GIL), and can saturate available CPU resources more quickly.

Key differences:

• The asyncio client usually uses less CPU per request and scales better when the client itself is not the performance bottleneck.
• The thread client can consume more CPU (reflected in CPU/wall metrics), and at high concurrency, the client-side overhead can distort latency measurements by adding client delays on top of server response times.

In short: Asyncio is event-driven and efficient for I/O-bound concurrency; thread clients use true OS threads and can incur higher per-request overhead, especially in Python.

Here are the test results:

The measurement-bias paper predicts that a saturated single-process asyncio client inflates TTFT and inter-token tails. This run did not put asyncio in that regime. The asyncio arm stayed at 0.24 to 0.28 CPU/wall. The thread arm sat at 1.10 to 1.24. TTFT p50 agreed within 3 ms both windows. TTFT p99 was higher on the saturated thread client both windows (1,412.5 vs 1,148.6, then 1,382.1 vs 745.5). Worst-gap p99 flipped direction across windows, so that delta is inside the noise floor.

The publishable claim is not “asyncio is faster” or “threads are faster.” The publishable claim is: the number that tells you which client is measuring itself is CPU/wall, not the runtime name, and a tail comparison that omits that number is not evaluable. At a few hundred milliseconds of server TTFT, concurrency 50 did not saturate a single asyncio event loop. It did push 50 OS threads past one core of client CPU.

If you need the paper’s M/G/1 regime, raise concurrency or QPS (queries per second, a measurement of how many requests your load generator is sending each second) until the asyncio arm’s own CPU/wall approaches 0.7, then compare.

How to read this chart. Same prompts, same 50 overlapping requests, two different programs sending them. Orange bars are the asyncio client. Blue bars are the OS-thread client. Left is the first window. Right is the repeat. Each group of bars is a different clock: typical wait for the first word (TTFT p50), unlucky-1-percent wait for the first word (TTFT p99), and unlucky-1-percent wait for the full answer (total p99).

The short pair on the left of each panel is the typical wait. Orange and blue are almost the same, within 3 ms. Most users would not feel a difference. The middle pair is the slow first-word case, and here the clients split: threads are slower both windows (1,413 ms vs 1,149 ms, then 1,382 ms vs 746 ms). The tall pair on the right is the slow full-answer case, around 10 seconds either way. What the bars do not show, and the table above does, is why: the thread client was burning more than one full core on itself (CPU/wall 1.10 to 1.24). The asyncio client was not (0.24 to 0.28). Do not read this as “asyncio is the faster server.” Read it as: the program that sends the requests can add delay of its own, and the typical wait will not warn you.

I ran this from a GPU Droplet. You do not need a GPU, though. A regular Droplet with 4 vCPUs is enough. That machine only sends the requests. The model lives somewhere else: DigitalOcean Serverless Inference, or a vLLM server you host on a GPU Droplet.

For Serverless Inference you need three things: a model access key, a prepaid balance, and the exact model ID from GET https://inference.do-ai.run/v1/models. If you host vLLM yourself, follow the continuous batching runbook and say whether prefix caching is on or off.

Python 3 is already on DigitalOcean images. The thread test needs nothing extra. The asyncio test needs pip install httpx.

Step 1. Create the machine that sends the requests

Spin up the Droplet. You can use our official doc on How to Create and Configure GPU Droplets.

Write down its region, size, and vCPU count. You will need those later when you publish. A number with no location is missing a piece the size of the delay.

Step 2. Get a key, then send one request

Put the key in the environment:

export BENCH_KEY=<your key>

List models first, the way the retrieve-available-models doc says. Then send one request. Do not start a 300-request run until you know the endpoint answers.

Step 3. Run the load sweep (Run A)

Download harness/protocol_demos.py script from the GitHub repo. Run the same mix of prompts at 1, then 8, then 50 overlapping requests, all with threads:

python3 protocol_demos.py --model <model-id> --arm threads --concurrency 1  --n 300 --out a_c1.json
python3 protocol_demos.py --model <model-id> --arm threads --concurrency 8  --n 300 --out a_c8.json
python3 protocol_demos.py --model <model-id> --arm threads --concurrency 50 --n 300 --out a_c50.json

Each command sends 310 requests and throws away the first 10. You keep 300. I ran this twice on the same morning and the unlucky-1-percent wait still moved 26 percent. That is why one 300-request p99 is not a finding on its own. The one-at-a-time cells are slow. Mine took 442 seconds and 460 seconds.

Step 4. Compare the two clients (Run B)

Now send the same 50 overlapping requests twice. Once with asyncio. Once with threads.

pip install httpx
python3 protocol_demos.py --model <model-id> --arm asyncio --concurrency 50 --n 300 --out b_async_c50.json
python3 protocol_demos.py --model <model-id> --arm threads --concurrency 50 --n 300 --out b_threads_c50.json

Look at client_cpu_seconds_per_wall_second in each summary. If that number is near 0.7 or above, the machine sending the requests is busy with itself. Put it next to the latency, not in a footnote. On my run, threads crossed that line. Asyncio did not.

Step 5. Do it again later

Repeat steps 3 and 4 in a different window. Overnight versus daytime is the honest pair. The gap between the two runs is your noise. If a “win” is smaller than that gap, it is not a win. If your second window disagrees with my tables, that disagreement is the finding. The endpoint is not a laboratory.

Step 6. Publish the files with the write-up

Put the JSON next to the article. The honest claim is not “the model is X ms.” It is “on this system, this change moved the number by Y, and here is everything I did.”

Every prompt in this harness starts with a unique prefix. vLLM and several hosted APIs reuse a repeated prefix by default. Without that unique start, you are timing the cache, not the model. If you want to time the cache, take the prefix out and say so. Either choice is fine. The quiet version is the trap.

The fifteen levers are the traps. This is how you close them.

The program that sends the requests. Use real threads or more than one process, unless you measure how busy that program is and it stays under about 0.7 CPU-seconds per wall-second. If the sender is busy, you are timing the sender. The fix is a second machine, not a louder burst. Timestamp on your machine: first visible word, the longest pause between words, and the last word. On this run, 50 threads crossed the line. 50 asyncio tasks did not.

How many requests, and how many times. I used 300 measured requests per cell, twice. Two one-at-a-time windows still differed 26 percent at the unlucky 1 percent and 5 percent at the typical wait. That gap is the noise I actually saw. Any “improvement” smaller than that gap is luck. Throw away the warm-up by prompt ID, and say how many you threw away. The Metrics that Matter in Serverless Inference article is where that rule comes from.

The mix of prompts. Do not send 300 copies of the same short question. Mix short, medium, and long. Write down the mix and the random seed so someone else can replay it. This harness uses 70 percent short, 20 percent medium, 10 percent long. Change the start of each prompt unless you are trying to measure the cache.

Write down the room you ran in. Model ID. Engine and image, if you can see them. Temperature. Token caps. Region. Where you sat relative to the server. What time it was. Keep every request’s JSON, not just the summary, so any number in the article can be rebuilt from the files. Background load changes through the day. One window is one weather report. Two windows is the minimum. You can check out the Why Serverless Inference Consistency Varies on the Same Model Study.

How to read this diagram. Each box is a lock on one way a number can cheat. Top row, left to right:

• A mixed pile of short, medium, and long prompts with a unique start so the cache cannot help.
• A sender that records how busy it is and stays under about 0.7.
• Clocks on your machine for first word, pauses, and last word
• A pinned room (model, engine, region, time window).
• Run the same test twice so the gap between runs is your noise, and keep the raw JSON so anyone can rebuild the percentiles. If a box is missing, that lever is still in the number.

Source: the live harness in this article, built from the protocols in the p50 vs p99 study, the Serverless Inference consistency study, and Serverless Inference Metrics that Matter.

Sit with this before you publish. Each box needs one sentence you could paste under a chart. If you cannot write that sentence, the number is not ready.

I filled mine in so you can see what “done” looks like. A copy you can tick lives in the GitHub repo. Version 1.0.

What did you time?

• [ ] 1. First word, or the whole answer? Write which, and whether the timer ran on your machine or on the server. This article: first visible word, timed on my Droplet.
• [ ] 2. How many requests? How many you kept, how many you threw away, and whether you ran it twice. This article: 310 sent, first 10 dropped, 300 kept. Ran twice. The slow 1 percent still moved 26 percent.
• [ ] 3. Where are the raw files? A link, or the sentence “I did not publish them.” This article: JSON in the GitHub repo.

Was the model already warm?

• [ ] 4. Did you drop the slow first calls? If yes, say how many. If no, say how many were still slow. This article: dropped the first 10 prompts I sent.
• [ ] 5. Could the cache help? Same prompt twice is a cache test, not a model test. This article: every prompt got a unique start so the cache could not reuse it.
• [ ] 6. Did you hide the slow first request on a fresh GPU? Dropping it is fine. Staying quiet is not. This article: those 10 dropped prompts cover the slow first hit.

How busy was the system?

• [ ] 7. One user, or a crowd? Write how many requests overlapped. This article: 1, then 8, then 50.
• [ ] 8. Did you only test the easy case? Show at least three loads, or write “I only tested one user.” This article: three loads, side by side.
• [ ] 9. How long did each run last? A 30-second test can miss a queue that blows up at minute three. This article: one-at-a-time cells took about 7 minutes. The second run started 12 minutes later, not overnight. That is a gap.

What did you send the model?

• [ ] 10. All short questions? Real traffic is mixed. Write the mix. This article: 70 percent short, 20 percent medium, 10 percent long, seed 7.
• [ ] 11. Temperature and token cap? Hidden settings make two tests incomparable. This article: temperature 0. Caps 100, 300, and 600 tokens.
• [ ] 12. Exact model name? The ID from the API, not a nickname. This article: mistral-3-14B. I could not see the engine version on serverless. That is a gap.

Who held the stopwatch?

• [ ] 13. Was your own machine busy? If CPU/wall is above 0.7, you timed the tester as well as the server. This article: at 50 overlapping requests, threads hit 1.02 to 1.24. Asyncio stayed at 0.24 to 0.28.
• [ ] 14. From where? Same datacenter as the API, or your laptop at home? This article: GPU Droplet in NYC2, same cloud as the API.
• [ ] 15. Did you keep only the lucky run? Publish every run, or write the drop rule before you look at the numbers. This article: both windows published. Neither was dropped.

If all 15 have an answer, someone else can rerun your test. If 1, 2, and 3 are blank, they cannot even check it. Empty boxes mean “trust me.”

I used this on my own table. Boxes 9 and 12 are the weak ones, and they stay in the article as gaps, not as something I papered over.

You should trust a benchmark as far as it tells you how it was made. Not as far as the logo on the page.

The fifteen levers name the ways a number moves. The live run showed a typical first-word wait rising 18 percent under load while the unlucky 5 percent nearly tripled and the wait for a full answer rose 3x. It also showed that the machine sending the requests can dirty the tail.

The setup above is how you stop that. The fifteen-box checklist is how a reader checks that you did.

If you want help deciding what to time, start with the p50 vs p99 latency study. If you want to run the same harness I ran, use this repo. If you want measured numbers under a stated protocol, read DigitalOcean’s LLM inference benchmarking piece. If you want why one number was always going to mislead, read the inference trilemma.

Then use the same fifteen boxes on every number you publish, and every number you read. The checklist is one page. Without it you are guessing.

Why 300 requests? Is that enough?

I used 300 kept requests per cell, twice. Two one-at-a-time windows still differed 26 percent at the unlucky 1 percent and 5 percent at the typical wait. That gap is the noise I actually saw. Any “improvement” smaller than that gap is luck, not a finding. Drop the warm-up by the prompt IDs you sent, not by which requests finished first, and write down how many you dropped. The Metrics that Matter in Serverless Inference article is where that rule comes from.

Why did you use threads instead of asyncio?

For the load sweep I used threads on purpose. At 50 overlapping requests the thread client burned 1.02 to 1.24 CPU-seconds per wall-second, so the slow tail mixes server delay with a busy sender. The asyncio client stayed at 0.24 to 0.28. Typical first-word wait agreed within 3 ms. The unlucky 1 percent did not. If you only ever run asyncio, you miss that trap. If you only ever run threads at high load and stay quiet about CPU/wall, you publish a dirty number. Use --arm threads or --arm asyncio in protocol_demos.py, and put the CPU/wall number next to the latency.

Do I need a GPU Droplet to run this?

No. The Droplet only sends the requests. A regular Droplet with 4 vCPUs is enough. I used a GPU Droplet in NYC2 because it was already sitting in the same cloud as the API. The model lives somewhere else: DigitalOcean Serverless Inference, or a vLLM server you host on a GPU Droplet.

Why not just publish the median?

Because the median can look fine while the unlucky people suffer. In window 1, the typical first-word wait rose 18 percent from one-at-a-time to 50 overlapping requests (202 ms to 238 ms). The unlucky 5 percent nearly tripled (445 ms to 1,245 ms). The wait for a full answer rose about 3x. If you only publish the blue line, load barely mattered. If you publish the orange and red lines, it did. Chat users feel the first word. Agents waiting on a full answer feel the whole thing. The p50 vs p99 latency study is the longer version of this answer.

How is this different from an LLM benchmark leaderboard?

Most “LLM benchmarks” score answer quality: MMLU, coding suites, Arena votes. This piece times how long the reply took to start and finish under load. Both matter. Confusing them is how a blog post about latency gets buried under a quality leaderboard, and how a quality win gets sold as “faster.” If you care about correctness, use a quality suite. If you care about whether your users wait, time the stream and fill the fifteen boxes.

I started this study because I kept reading latency numbers I could not rerun. A clean chart. A winning setup. No way to tell whether the author timed the model, the cache, or a lucky morning.

So I ran the test myself. Same prompts, same machine, twice. The slowest 1 percent still moved 26 percent. Load barely touched the typical wait and nearly tripled the unlucky 5 percent. At 50 overlapping requests, the thread client was busier than one full core, and that dirt showed up in the tail. None of that is exotic. It is what happens when you keep the raw files and look. It is why this study is different from other benchmarks.

The fifteen boxes at the end are the part I want you to keep. Tick them before you publish. Hold them up when someone else publishes.

The harness, the raw JSON, and the charts live in the GitHub repo. Use them, disagree with them, and publish your own gaps next to your own numbers. That is the whole point.

DigitalOcean documentation and engineering content

• LLM Inference Benchmarking: Measure What Matters (DigitalOcean’s measured results, the numbers-led complement to this piece)
• p50 vs p99 Latency: Why Median Benchmarks Mislead AI Agent Workloads (the consumer-side argument and the reference harness)
• Metrics that Matter with Serverless Inference (trial-count, warm-up, and multi-window protocol)
• Why Serverless Inference Consistency Varies on the Same Model (methodology-forward publishing pattern; precision opacity findings)
• How Does Prompt Caching Work: The Cost Break-Even (the cache-state lever, quantified)
• Prompt Caching in Practice: From 7% to 74% Hit Rate (the same lever in production traffic)
• When Your vLLM p99 is Worse Than Your p50 (the concurrency lever’s mechanism)
• Why is p99 TTFT High When Everything Else Looks Normal? (operational debugging of the same tail)
• The Hidden Bottlenecks in LLM Inference (serving-side levers: batching, KV cache, prefill)
• Ornith 9B Fine-Tuning Cost Benchmark (a documented first-hit artifact, disclosed honestly)
• Choosing the Right Model for Your Inference Use Case (the quality-benchmark lever set, out of scope here)
• The LLM Inference Trilemma (why single-number benchmarks structurally mislead)
• Token Economics on Dedicated GPUs (utilization context for throughput disclosure)
• The Data Locality Tax (the network-position lever, priced by physics)
• Use Serverless Inference
• How to Create and Manage Model Access Keys
• How to Retrieve Available Models
• Chat Completions API
• Inference Pricing

Evidence repositories

• anishsingh20/inference-benchmark-design-standard (this piece: harness, raw JSON, charts, disclosure standard v1.0)
• anishsingh20/serverless-inference-tail-latency-study (p50 vs task-time inversion on the same endpoint family)
• anishsingh20/data-locality-tax (measured cross-region retrieval floors)

External sources

• Ashok Chandrasekar and Jason Kramberger, “Identifying and Mitigating Systemic Measurement Bias in Production LLM Inference Benchmarks,” arXiv, 2026 (client-side M/G/1 queueing bias)
• Artificial Analysis, Language Model Benchmarking Methodology
• vLLM Optimization and Tuning documentation (default-on prefix caching and scheduler behavior)
• MLCommons MLPerf Inference