Reading DeepInfer Benchmarks: V4.1-Flash Performance

DeepInfer's V4.1-Flash benchmarks are easy to quote and hard to interpret. A tokens-per-second number is not a property of an engine; it is a property of an engine, a model, a batch, a context length, a hardware configuration, and a workload. The chart is a map of one journey, not a guarantee about yours.

The units trap

Three different rates appear in serving discussions, and they are not interchangeable:

A system can have high tokens/s/GPU and low tokens/s/user at the same time. That is exactly what the decode chart shows: as concurrency rises, aggregate throughput rises while each user's stream slows down. Neither number is “the speed” of the engine.

End-to-end latency adds network, queueing, tokenization, scheduling, and API overhead. None of those are in either chart. A 100 ms/step decode figure does not mean a 100 ms time-to-first-token; a 325 tokens/s/user figure does not mean a 325 ms response.

Prefill: batch buys throughput, latency pays for it

Prefill processes the prompt and builds the KV cache. Larger batches amortize the weight reads and improve GPU utilization, so throughput rises. But each step also takes longer, so latency rises.

DeepInfer's prefill chart uses 4 SM100 GPUs at a 131,072-token context. The labels are batch sizes from 4,096 to 65,536. The curve rises steeply at first, then flattens:

• At 4,096 tokens per batch, throughput is around 60k tokens/s/GPU and each step takes about 68 ms.
• At 16,384, throughput is around 81k and each step is about 200 ms.
• At 65,536, throughput is around 89k and each step is about 730 ms.

The knee matters more than the peak. If your SLO is tight, you live on the left side of the curve and pay in throughput. If you are running offline batch evaluation, you can live on the right side and accept longer steps. The chart does not tell you which side is correct — your workload does.

Also note what the prefill chart is measuring: non-cached input throughput. If most of your prompt is a cache hit, the prefill work is much smaller, and this curve is not your bottleneck. Cache hit rate is a separate variable, and it is the variable the missing KV cache system is designed to maximize.

Decode: concurrency buys throughput, interactivity pays for it

Decode generates one token per request per step. With more concurrent requests, the GPU does more useful work per step, so aggregate throughput rises. But each request waits longer between tokens, so per-user speed falls.

DeepInfer's decode chart uses 64 SM100 GPUs at a 131,072-token context, with nextn=3 fixed and 75% per-token draft acceptance. The labels are requests per DP rank:

• At 8 requests per rank, aggregate throughput is around 2.6k tokens/s/GPU and each user sees about 325 tokens/s.
• At 64 requests per rank, throughput is around 15.5k and each user sees about 248 tokens/s.
• At 256 requests per rank, throughput is around 26k and each user sees about 103 tokens/s.

The shape is the classic serving trade-off. A chat product might choose the left side; a batch summarization or offline agent workload might choose the right. The chart is useful because it shows the whole frontier, not a single point.

The example README gives the accounting formula behind the decode throughput number:

decode throughput = avg_SPS × decode_num_requests × (1 + nextn) / cp

This is “full-acceptance accounting”: it assumes every draft token is accepted, regardless of the true acceptance rate. The chart scales the result by a 0.75 acceptance assumption. If your text is harder to predict, the real number is lower. If it is easier, it could be higher.

The 75% assumption

Speculative decoding proposes draft tokens and verifies them in a batch. The gain depends on how many drafts are accepted. The decode chart assumes a 75% per-token acceptance rate, which yields an average accepted length of about 3.25 tokens per step.

That assumption is plausible for predictable text and less plausible for open-ended reasoning, code with unusual identifiers, or highly technical writing. DeepSeek's own production data shows the accepted length around 3.2–3.3 for its DSpark configuration, so the chart's assumption is not arbitrary — but it is still an assumption, not a property of your workload.

The related DSpark optimization — variable-length verification — is also load-dependent. DeepSeek reports +4–7% throughput at its production point, but a small loss at low concurrency. This is a good example of why “speculative decoding is free” is the wrong mental model: the gain depends on acceptance and on how much verification capacity you have to spare.

What the eval numbers measure

The eval scripts are not a serving benchmark. They measure model quality on three offline tasks:

• AIME 2026: 30 tasks, 32 samples each, Pass@1 99.9%.
• GPQA Diamond: 198 tasks, 8 samples each, Pass@1 93.4%.
• HLE-Full: 2,500 tasks (2,158 text, 342 image), 1 sample each, Pass@1 42.6%.

Pass@1 is the probability that a single sample is correct. With 32 samples per task, AIME's 99.9% is an average over 960 rollouts; HLE's single sample is much noisier. The eval README warns that repeated runs can differ and that a Pass@1 figure is only valid when coverage_complete is true.

These numbers tell you that the model is capable on these benchmarks under the default configuration. They do not tell you that the engine is faster, cheaper, or more reliable than another engine. For that, you need a serving benchmark with the same hardware, context, and workload — and ideally a third party running it.

A reading checklist

When you see an inference benchmark, ask:

1. What is the unit? tokens/s/GPU, tokens/s/user, or ms/step?
2. What is the workload? Context length, input/output ratio, cache hit rate, request arrival pattern.
3. What is the hardware? GPU/NPU model, count, interconnect, memory, and whether it is a single node or a cluster.
4. What is the batch/concurrency? A throughput peak at high concurrency may be irrelevant to a latency-sensitive product.
5. What is included? Engine only, or the full serving stack with KV cache, routing, and API overhead?
6. What is assumed? Draft acceptance, cache hit rate, quantization, tensor/pipeline parallelism.
7. Who ran it? Vendor self-measurement, independent lab, or a reproduction with a commit hash?
8. What is the comparison? Same model, same hardware, same workload, same SLO?

If a claim fails three or more of these, treat it as a directional signal, not a procurement number.

Back to the chart readout, or on to the missing service layer.