Inside DeepInfer: DeepSeek's V4.1-Flash Architecture

DeepInfer is DeepSeek's open-source inference architecture for V4.1-Flash. To read it well, separate three layers: the model, the engine, and the service. The model is V4.1-Flash. The engine is what this page is about. The service is the part DeepSeek kept.

What an inference engine actually does

A trained model is a pile of weights. Serving it means answering requests: read the prompt, run the model, generate tokens, return them, and do that for many users at once without wasting the expensive hardware.

Two phases dominate the cost:

• Prefill reads the prompt and builds the KV cache. It is compute-heavy and can process many prompt tokens in parallel.
• Decode generates one token at a time. Each step reads the KV cache and a slice of the model; it is memory-bandwidth-heavy and sensitive to per-step latency.

Most serving optimizations are attempts to make one of those phases cheaper, or to overlap it with something else. DeepInfer's features map cleanly onto that split.

The deployment shape: EPD, Wide-EP, and a lot of GPUs

DeepSeek's Q&A describes the production V4.1 system as an Vision-Encoder / Prefill / Decode (E/P/D) deployment:

The split exists because prefill and decode want different hardware shapes. Prefill is compute-bound; decode is memory-bound. Putting them on the same pool forces a compromise. Separating them lets each stage scale independently — at the cost of moving KV cache between pools and balancing two different workloads.

DeepSeek's production numbers:

DeepSeek calls the smallest efficient unit 64 GPUs of prefill plus 64 GPUs of decode. A 1P + 1D setup is deployable, but the recommended decode layout is EP64 or above, so small clusters will not see the same efficiency.

Expert load balancing: redundant copies and periodic reshuffling

MoE routing is uneven. Some experts get hot; others idle. A static assignment wastes GPUs. DeepSeek's EPLB (expert-parallel load balancing) adds redundant experts — replicas of hot experts — and periodically reassigns experts to GPUs based on recent load.

The mechanism, as described in the Q&A:

• Each GPU holds its normal experts plus a few redundant experts.
• The system records each expert's load over the last N steps.
• Every M steps, it reassigns experts to GPUs and moves weights within the NVLink domain using symmetric memory.

moe_lb is the slowest rank divided by the mean. A value of 1.09 means the slowest GPU does about 9% more work than average. At decode scale, that is a tight number; it also means the fastest GPU still waits at the barrier.

DSpark: spend verification compute where it pays

Speculative decoding uses a draft model to propose several tokens, then verifies them in one pass. If the draft is right, you get multiple tokens for the cost of one verification. If it is wrong, you fall back.

The naive version verifies a fixed number of drafts. DSpark's twist is variable-length verification: a confidence head predicts the acceptance probability of each draft token, and the scheduler chooses the prefix subset that maximizes effective throughput.

DeepSeek also publishes DeepSpec, an MIT-licensed codebase for training and evaluating speculative-decoding draft models. It covers DSpark, DFlash, and Eagle3; it is separate from the DeepInfer engine.

DeepSeek's production measurements:

The interpretation matters: verification work drops by roughly 30%, while accepted length barely moves. The net gain is positive at high concurrency. At 16 sequences per rank, the table shows a small loss (−1.5%). This is not a free lunch; it is a load-dependent optimization.

The default maximum draft length in the release is 5. The published decode chart, however, uses nextn=3 fixed and a 75% per-token acceptance assumption — a measurement configuration, not the default deployment.

SWA bounded replay: make a miss cheap, then stop storing the cache

Sliding-window attention (SWA) keeps only the most recent tokens in full attention. V4.1 uses a 128-token window. If the SWA KV cache is missing, exact recovery is expensive: each decoder layer's window depends on the layer below, so rebuilding the state across 20 decoder layers costs roughly 2,560 token-layer replays (20 × 128).

SWA bounded replay does something deliberately approximate:

• Replay only the last 128 tokens.
• Truncate the attention window at the replay start: position i attends only to [max(s, i−127), i].
• Accept an approximate SWA KV cache, with a bounded, constant recompute cost.

The release's default V4.1 setting keeps a 140-token suffix (DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140). The extra room covers the window plus speculative backtracking.

Why this matters beyond saving compute: it makes it rational not to persist SWA KV. V4.1's persistent KV cache therefore stores global KV, not SWA KV, and shrinks to roughly one-eighth of V4's footprint. Global KV itself is compressed to about 890 bytes per token, roughly one-quarter of V4-Flash.

Asynchronous scheduling: keep the GPU out of the CPU's way

A decode step looks like this:

schedule(N) → forward(N) → sample(N) → post-process(N)

schedule and post-process are CPU work. Before post-process(N), the CPU needs the result of sample(N) from the GPU. That synchronization leaves the GPU idle while the CPU processes the result and schedules the next batch.

DeepInfer overlaps the two:

GPU: forward(N) → sample(N)
CPU: schedule(N+1) → launch_forward(N+1) → post-process(N−1) → launch_sample(N+1)

The idea is not new — DeepSeek cites SGLang and NanoFlow — but the implementation is part of the release and is enabled by default. It is a good example of the kind of work that separates a benchmark kernel from a serving engine.

Batch-invariant inference and deterministic sampling

Dynamic batching changes the order of floating-point reductions. The same prompt can produce slightly different logits depending on which other requests happened to share the batch. For chat, that is usually invisible. For benchmarking, auditing, or reproducing an agent run, it is a problem.

DeepInfer exposes two levels of control:

• Batch-invariant inference (DEEPINFER_BATCH_INVARIANT=1): results stay consistent across different batch compositions.
• Deterministic sampling: identical input tokens and sampling parameters produce identical output. It requires batch-invariant mode, MTP variable-length decoding off (DEEPINFER_ENABLE_MTP_VARLEN=0), and a fixed seq_seed.

Both are NVIDIA-only in this release. They are not headline features, but they are the kind of thing a serious evaluation team notices.

Structured outputs: constrain the grammar, not the prompt

Structured output constrains generation to a regex, a JSON Schema, or an EBNF grammar. DeepSeek's production design compiles the constraint into a grammar automaton on a separate machine, then applies a token mask at each decode step. Compilation and inference are decoupled, so a request can pick up a compiled grammar before it enters the engine.

In the release, the API is SamplingParams(so_args={"schema_type": ..., "schema": ...}). The offline example can also run regex, JSON, and EBNF checks with --check_structured_output and verify normal EOS termination.

This is not unique to DeepInfer — constrained decoding is a well-known technique — but having it inside the serving engine is convenient for agent and extraction workloads.

DeepSeek Recipe: the other half of “API-compatible”

An inference engine computes tokens. An API service also has to:

• accept Chat Completions, Responses, and Messages requests;
• render the model's prompt format, including thinking mode, tools, and images;
• encode text into token IDs without letting user text inject special tokens;
• parse streaming output across chunk boundaries;
• turn reasoning content, tool calls, and stop sequences back into protocol responses.

deepseek-recipe does that. It is a separate MIT-licensed repository: Rust libraries with Python bindings, plus example servers. DeepSeek says its processing logic is aligned with the official API, so third-party deployments can behave like the official service.

Two details are worth appreciating:

• Special-token injection defense. If a user types <｜User｜> or <｜image｜>, Recipe encodes it as ordinary text, not as a control token. The model cannot be tricked into treating user content as a role boundary.
• Segmented parallel tokenization. Special tokens and their positions are known when the prompt is built, so the ordinary-text segments between them can be encoded independently and concatenated. That reduces tokenization latency for long contexts.

If you are evaluating DeepInfer as a serving stack, Recipe deserves as much attention as the engine. The engine is the expensive part; Recipe is the part that makes the API feel right.

New idea or old idea?

The honest framing: DeepSeek did not invent most of these primitives. The contribution is the integration — and the willingness to publish the production tuning.

Production design ≠ open-source code

What the Q&A does give you is a rare map of the decisions behind the code: why Wide-EP, why 64+64, why 3FS, why a 3/7-day TTL, why bounded replay, why variable verification. That map is the most valuable part of the release for an engineer at another lab — even if the code stops at the engine boundary.

Next: how to read the two charts, what the missing pieces mean for deployment, or the serving comparisons: vLLM, SGLang, TensorRT-LLM.