DeepInfer Production Gap: KV Cache, PD, and Load Balancing

DeepInfer is DeepSeek's V4.1-Flash inference engine, not a complete serving stack. The README's “not included” list is short: a KV cache system, VE/Prefill/Decode disaggregation, and load balancing across data-parallel replicas. Each item is a subsystem, not a feature flag. Together they are the difference between running a model and running a service.

Engine vs service

An engine takes a batch of tokenized requests and returns generated tokens. A service takes HTTP requests from many users, authenticates them, routes them, keeps their context warm, enforces quotas, handles failures, and bills them. The engine is the expensive compute; the service is the expensive operations.

DeepInfer is the engine. DeepSeek's Q&A describes the service around it in unusual detail, which makes the gap easy to see. The rest of this note walks through the three biggest pieces.

KV cache: the memory that makes long contexts affordable

When a model reads a prompt, it stores keys and values for every token and layer. That cache is what lets decode generate the next token without re-reading the whole prompt. Reusing a prefix across requests is one of the largest cost levers in serving: a long system prompt, a shared document, or a multi-turn agent history can be computed once and reused many times.

The naive version — keep a dictionary in GPU memory — collapses at scale. GPU memory is small relative to demand, a cache entry may be needed by a different machine, and compute nodes fail more often than storage nodes. DeepSeek's production design separates three tiers:

The production numbers are large enough to change how you think about the problem:

• About 100 PiB of SSD for KV cache.
• About 14.4 TB of logical KV per GPU; physical capacity is larger.
• About 400 Gbps of interconnect per 8 GPUs.
• 3FS with 10+2 erasure coding; each EC group tolerates two simultaneous node failures.
• Peak aggregate read around 10 TiB/s; write around 850 GiB/s.

Why a dedicated storage cluster instead of local memory or a tiered cache? Three reasons from the Q&A:

1. Any prefill machine can hit any prefix. If cache were local, routing would need prefix affinity, which conflicts with load balancing.
2. Storage and compute scale independently. You can add cache capacity without adding GPUs, and vice versa.
3. Compute failures do not lose the cache. Compute nodes fail more often than storage nodes by an order of magnitude, according to DeepSeek.

A cache hit has three stages: match, fetch, and swap-in. Only the match is on the critical path; fetching from 3FS to host shared memory happens asynchronously in a separate process, and swap-in from shared memory to HBM is overlapped layer by layer with forward computation. DeepSeek uses GDR/RDMA for the swap-in because the CUDA copy engine is inefficient for fine-grained KV block copies.

If you had to build this yourself, you would need prefix matching, a metadata service, a distributed store, eviction/TTL, cross-machine transfer, fault tolerance, and a routing policy that cooperates with load balancing. That is a storage team's worth of work, not a weekend project.

Prefill/decode disaggregation: two workloads, two pools

Prefill and decode have different bottlenecks. Prefill is compute-bound: it processes many prompt tokens at once and benefits from large matrix multiplications. Decode is memory-bandwidth-bound: each step generates one token per request and reads the KV cache. Putting them on the same pool forces a compromise in hardware, parallelism, and batch size.

Disaggregation splits them into separate pools:

• Prefill pool: 8-GPU machines, EP2–8 / DP2–8, sized by non-cached tokens.
• Decode pool: 64–128-GPU supernodes, EP64–128 / DP64–128 (Wide-EP), sized by running requests.
• Vision encoder: one GPU per instance, often a cheaper card.
• Storage: dedicated KV cache nodes, separate from compute.

DeepSeek says the P:D ratio is roughly 1:1 and calls 64 prefill + 64 decode GPUs the smallest efficient unit. A 1P + 1D setup is deployable, but the recommended decode layout is EP64 or above.

The hard part is the handoff. A request arrives, gets prefilled, produces KV cache, and then must be routed to a decode engine that can see that cache. That requires a transfer path, a directory of where the cache lives, and a routing decision that accounts for both cache location and decode load. DeepSeek budgets roughly 400 Gbps of interconnect per 8 GPUs and notes that P and D can be in adjacent data centers, at some reliability cost.

The release says “most required engine functionality is already implemented” for disaggregation, but the full scheme is not included. If you want the production shape, you are building the orchestration layer.

Load balancing: two dimensions, four balancers

DeepSeek describes four balancers: prefill, decode, vision encoder, and expert-parallel.

Prefill load balancer

Requests are bucketed by total context length into four tiers: <16K, 16–64K, 64–256K, and ≥256K. The number of engines assigned to each tier is redistributed based on the previous round's compute-quota demand. Within a DP group, each request goes to the rank with the lowest current quota load. Within a GPU, requests are dequeued FIFO, and the chunk size is solved from a time budget using a fitted quadratic cost model.

This is more sophisticated than round-robin because prefill cost is not linear in request count; a single 256K-token prompt can dominate a batch of short ones.

Decode load balancer

Decode routing balances two dimensions: request count (which determines per-step batch size) and total KV cache length (which determines attention cost). The heuristic sends short requests to GPUs with low concurrency but large KV length, and long requests to GPUs with small KV length. The goal is to keep both the batch and the attention work balanced.

Expert-parallel load balancer

EPLB adds redundant experts and periodically reassigns experts to GPUs based on recent load. DeepSeek reports decode moe_lb around 1.090 — the slowest rank does about 9% more work than average. Prefill numbers differ slightly between the English Q&A and the Chinese post, but the mechanism is the same.

The 80% reality check

The most useful sentence in the Q&A might be this: production load balancing reaches roughly 80% of a perfectly balanced offline benchmark.

That means even DeepSeek, with custom balancers and full control of the stack, reaches only about 80% of the offline benchmark's throughput. A team bolting a simple router onto DeepInfer should not expect better. It also means the offline Pareto charts are upper bounds: they are measured in a controlled setup, not under production routing skew.

This is not a criticism. It is a calibration: distributed serving is messy, and the last 20% is where much of the engineering lives.

Build vs buy

If you need a served API, you have three broad paths:

For most teams, the first path is the pragmatic one. DeepInfer is for teams with the hardware, the workload, and the engineering capacity to own the service layer — or for engineers who want to learn how DeepSeek does it.

Back to the limits page, or read how the engine works.