The paper is the receipt. Don't file 27% FLOPs or 10% KV as your serving bill.

Buyers who only compare HuggingFace cards will miss the infrastructure half of the paper: MegaMoE, TileLang fused kernels, batch-invariant decoding, hybrid ZeRO buckets, and on-disk shared-prefix KV. Those sections are why a local bench can miss the Figure 1 ratio even when the architecture matches.

This Record reads DeepSeek-V4's efficiency claims as written in arXiv:2606.19348. The ratios are architectural estimates at a filled 1M-token context versus DeepSeek-V3.2, not a token invoice from your cluster. Pro is 1.6T total and 49B activated. Flash is 284B total and 13B activated. Both ship MIT weights on HuggingFace. Monday use still needs a local trace: your context-length mix, your precision, your kernel.

If the only number you take from the paper is the headline ratio, you will under-buy decode capacity when CSA top-k, sliding-window tails, and heterogeneous KV layout stop matching the plot. Keep the figure. Don't skip the methods.

DeepSeek-AI, "DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence" (arXiv:2606.19348, posted April 26, 2026), releases preview checkpoints of two Mixture-of-Experts (MoE) models: DeepSeek-V4-Pro (1.6T total parameters, 49B activated per token) and DeepSeek-V4-Flash (284B total, 13B activated): both natively trained for one-million-token contexts. The paper's central efficiency claim is architectural: at a filled 1M-token context, V4-Pro requires 27% of the single-token inference FLOPs and 10% of the accumulated KV cache size relative to DeepSeek-V3.2, despite activating more parameters than V3.2 in the Pro variant. V4-Flash pushes further to roughly 10% FLOPs and 7% KV cache (Figure 1, Abstract, §Introduction). The mechanism is a hybrid attention stack: Compressed Sparse Attention (CSA) interleaved with Heavily Compressed Attention (HCA): plus precision and infrastructure work on MoE communication, deterministic kernels, and heterogeneous KV management. Checkpoints ship under MIT license on HuggingFace. This Record piece reads the efficiency claims as stated in the paper; wall-clock dollars per million agent tokens on your stack remain UNKNOWN until benchmarked locally.

Claim of novelty

Hybrid CSA + HCA attention (§2.3, Figure 2). DeepSeek-V4 retains the DeepSeekMoE feed-forward pattern (Dai et al., 2024) and Multi-Token Prediction (MTP) from DeepSeek-V3 (DeepSeek-AI, 2024c), but replaces vanilla attention with two compressed mechanisms used in alternation across most layers.

Compressed Sparse Attention (CSA, §2.3.1, Figure 3). CSA first compresses KV along the sequence dimension: every m token positions produce one compressed entry via softmax-gated pooling over overlapping windows of learnable KV series Ca and Cb with positional biases (Eqs. 9–12). Sequence length drops by factor m. CSA then applies DeepSeek Sparse Attention (DSA) lineage (DeepSeek-AI, 2025b): a lightning indexer scores compressed blocks with low-rank queries and ReLU-gated multi-head dot products (Eqs. 13–16), selects top-k compressed entries per query (Eq. 17), and runs shared-KV multi-query attention (MQA) on the sparse subset (Eqs. 18–19). A sliding-window branch adds n_win uncompressed recent tokens so queries can see fine-grained local context inside the current compression block (§2.3.3). Grouped output projection splits large head stacks to reduce projection FLOPs (§2.3.1).

Heavily Compressed Attention (HCA, §2.3.2, Figure 4). HCA uses a much larger compression ratio m′ ≫ m without sparse selection: every m′ tokens merge to one compressed entry via softmax weights (Eqs. 20–23), then dense MQA runs over the short compressed sequence. HCA trades extreme KV shrinkage for full attention over the condensed stream: feasible because the condensed length is small.

Layer schedule (§4.1 configuration tables). DeepSeek-V4-Pro uses 61 Transformer layers, hidden size 7168: layers 0–1 are pure HCA; layers 2–60 alternate CSA and HCA; MTP tail uses sliding-window attention only. Pro CSA: m = 4, indexer heads n_h^I = 64, indexer dim 128, sparse top-k = 1024, HCA m′ = 128, query heads n_h = 128, head dim c = 512, query compression d_c = 1536, sliding window n_win = 128. DeepSeek-V4-Flash uses 43 layers, d = 4096, with CSA top-k = 512 and otherwise parallel ratios (m = 4, m′ = 128). First MoE layers in both models use Hash routing (Roller et al., 2021) instead of learned gating; Sqrt(Softplus) replaces Sigmoid on router scores versus V3.

Manifold-Constrained Hyper-Connections (mHC, §2.2). mHC expands the residual stream width n_hc and constrains the residual map B_l to doubly stochastic matrices via Sinkhorn-Knopp iteration (Eqs. 1–8), bounding spectral norm for stack stability: addressing numerical blow-ups seen in naive Hyper-Connections (Zhu et al., 2025). Dynamic plus static parameterization generates A_l, B_l, C_l mappings (Eqs. 3–7). Paper cites dedicated mHC work (Xie et al., 2026).

Muon optimizer (§2.4, Algorithm 1). Muon with hybrid Newton-Schulz orthogonalization replaces AdamW on most modules; AdamW remains on embeddings, prediction head, mHC static biases, and RMSNorm weights. Authors report faster convergence and stability; QK-Clip omitted because RMSNorm on queries and KV entries controls logit scale.

Infrastructure novelty (§3). Beyond attention math, the paper claims production-grade enablers: fine-grained MoE expert-parallel mega-kernel overlapping dispatch/combine with linear stages (§3.1, MegaMoE in DeepGEMM); TileLang fused kernels with host codegen and Z3-assisted bound analysis (§3.2); batch-invariant dual-kernel attention decoding and deterministic backward paths for sparse attention and MoE (§3.3); hybrid ZeRO buckets for Muon (§3.4.1); two-stage contextual parallelism for compressed attention across CP ranks (§3.4.3); heterogeneous KV layout with classical compressed blocks plus state cache for SWA tails and unready compression buffers (§3.5.1, Figure 6); on-disk shared-prefix KV with hierarchical caching (§3.5.2).

Efficiency framing versus baselines (§2.3.4, Figure 1). Authors compare accumulated KV and per-decode-step FLOPs against DeepSeek-V3.2 and against BF16 GQA-8 with head dim 128, reporting ~2% of that GQA KV at 1M tokens for V4 series. FP8 storage on non-RoPE KV dims, FP4 indexer math in CSA, and smaller sparse top-k versus V3.2 on short contexts compound the headline ratios.

Post-training (§Introduction, §4). Two-stage pipeline: domain experts (math, code, agent, instruction) via SFT then GRPO RL, consolidated by on-policy distillation into unified models. FP4 quantization-aware training on MoE expert weights and indexer QK paths reduces memory in post-training (§Introduction).

What was measured and on which data

Primary efficiency metrics (Abstract, Figure 1, §2.3.4). The paper's efficiency evaluation is analytic and architectural rather than a serving trace replay. At 1M context length, authors plot estimated single-token inference FLOPs (equivalent FP8 FLOPs) and accumulated KV cache bytes for V3.2 versus V4-Pro and V4-Flash. Reported ratios: V4-Pro at 27% FLOPs and 10% KV versus V3.2; V4-Flash at ~10% FLOPs and ~7% KV versus V3.2. Routed experts use FP4 weights; paper notes current hardware peak FP4×FP8 equals FP8×FP8 but future hardware could improve FP4 efficiency by ~33%.

Pre-training scale (§Introduction, §4). V4-Flash pre-trained on 32T tokens; V4-Pro on 33T tokens. After pre-training both support 1M contexts natively. Internal base-model comparisons: V4-Flash-Base beats V3.2-Base on a majority of benchmarks; V4-Pro-Base leads across reasoning, coding, long-context, and knowledge suites per authors.

Standard benchmarks: Pro-Max vs frontier (Table 6, §5.3.2). DeepSeek-V4-Pro-Max (maximum reasoning effort) compared to Opus 4.6, GPT-5.4, Gemini 3.1 Pro, Kimi K2.6, GLM 5.1. Knowledge: MMLU-Pro EM 87.5 (second to Gemini 91.0); SimpleQA-Verified Pass@1 57.9 (leading open, below Gemini 75.6); Chinese-SimpleQA 84.4; GPQA Diamond 90.1; HLE 37.7. Code: LiveCodeBench 93.5; Codeforces rating 3206 (23rd among humans on internal contest sim). Math: HMMT 2026 Feb 95.2; IMOAnswerBench 89.8; Apex Shortlist 90.2. Long 1M: MRCR 1M MMR 83.5 (below Opus 92.9, above Gemini 76.3); CorpusQA 1M ACC 62.0 (above Gemini 53.8). Agent: Terminal Bench 2.0 67.9%; SWE-Verified 80.6%; SWE-Pro 55.4%; BrowseComp 83.4%; MCPAtlas Public 73.6%; Toolathlon 51.8%. Authors left some competitor API cells blank when services were unavailable.

Within-family modes (Table 7). V4-Flash and V4-Pro across Non-Think, High, and Max reasoning modes. Max mode uses longer contexts (up to 384K on reasoning tasks) and wins on hardest suites: e.g., V4-Pro-Max MRCR 1M 83.5 vs Non-Think 44.7; HLE 37.7 vs 7.7 Non-Think. Figure 10 plots reasoning effort vs cost on HLE and Terminal Bench versus V3.2: V4 improves token efficiency on HLE relative to V3.2 when scaling test-time compute.

1M retrieval curves (Figure 9, §5.3.2). MRCR performance stable through 128K; degradation beyond 128K but strong at 1M versus peers. CorpusQA framed as more realistic long-document QA than synthetic MRCR needle tests.

Agent harness details (§5.3.1). Code agents: internal framework with bash and file-edit tools, 500 steps, 512K context cap. Terminal-Bench 2.0 Verified subset ~72.0% for V4-Pro per footnote. Search agents: websearch plus Python tools, discard-all context management on BrowseComp as in V3.2. Formal math: Lean 4.28 agent with tactic search up to 500 tool calls; hybrid informal-to-formal pipeline for Putnam-2025 frontier regime (Figure 8).

Infrastructure microbenchmarks (§3.1). Fine-grained EP mega-kernel: 1.50–1.73× vs non-fused baselines on general inference; up to 1.96× on RL rollout / agent-serving latency cases. Compute-communication balance threshold cited: C/B ≤ 2d FLOPs/byte (6144 for d=3072-class hidden sizes) for full overlap.

Real-world internal eval (§5.4). Chinese functional writing win rate 62.7% vs Gemini 3.1 Pro (Table 12); creative writing 60% instruction following / 77.5% quality vs Gemini (Table 13); complex multi-turn writing still trails Claude Opus 4.5 (Table 14). RAG search and agentic search beat V3.2 on internal Q&A tables. White-collar Chinese professional tasks: human eval vs Opus 4.6-Max: 63% non-loss rate (Figure 12). Internal R&D coding benchmark: 30 filtered tasks from ~200 engineer submissions: V4-Pro approaches Opus 4.5 (Table 8).

Buyer reading guide for efficiency figures. Figure 1 right panel is the headline efficiency evidence: read KV and FLOPs curves as context length grows, not single-point marketing at 8K. Table 6–7 separate quality from efficiency; a platform can adopt V4 weights without achieving 1M context economics if serving stack ignores compressed KV layout. §2.3.4 GQA-8 baseline clarifies KV is ~2% of conventional GQA8-BF16 at 1M: useful when comparing to vLLM paging on standard attention checkpoints.

Attention sink and RoPE partials (§2.3.3). CSA and HCA use learnable attention sink logits per head so total attention mass need not normalize to unity: stabilizing very long compressed sequences (Xiao et al., 2024; OpenAI, 2025). Partial RoPE applies to last 64 dimensions on queries, KV entries, and core attention outputs with counter-rotation so outputs carry relative position information despite compressed keys serving as both key and value.

FP4 QAT on experts (§5.2 area). Post-training employs FP4 quantization-aware training on MoE expert weights with lossless FP4-to-FP8 dequantization because FP8 exponent range absorbs sub-block scale ratios: allowing reuse of FP8 training pipelines. Indexer QK paths also FP4-trained. Buyers fine-tuning experts should verify QAT assumptions hold on adapter deltas.

DSec agent sandbox (§5.2). Paper describes DSec execution substrates (function call, container, microVM, fullVM) with trajectory logging, preemption-safe replay, and 3FS-backed image layers: supporting agent RL rollouts that stress long contexts. Links agent evaluation infrastructure to million-token training loops.

Compression parameter buyer table (§4.1).

VariantLayersdCSA mHCA m′CSA top-kn_win
V4-Pro61716841281024128
V4-Flash4340964128512128

Why interleave CSA and HCA. CSA preserves query-dependent sparsity on moderately compressed blocks; HCA supplies cheap global context on heavily compressed streams. Uniform attention type across all layers would waste capacity: HuggingFace blog and paper §2.3 motivate alternation.

Test-time scaling link (Introduction). Authors explicitly tie million-token efficiency to reasoning models and agent horizons: quadratic vanilla attention blocks ultra-long rollouts; V4 efficiency claims are the enabling arithmetic for longer thinking traces without proportional KV growth.

License and open-weight posture. MIT-licensed checkpoints on HuggingFace differentiate V4 from several frontier APIs: buyers can inspect weights, run air-gapped eval, and pin revisions. Export and deemed-export rules for large models remain buyer legal review items not addressed in the paper.

Baseline fairness

Same lineage comparison for efficiency. FLOPs and KV ratios versus DeepSeek-V3.2 isolate attention and storage innovations within one training and tokenizer family: fair for architectural efficiency claims, not for cross-vendor serving cost leaderboards.

V3.2 already sparse. V3.2 introduced DSA; V4's CSA runs sparse selection on pre-compressed blocks: apples-to-apples efficiency comparison against an efficient predecessor, not vanilla MHA.

Activated-parameter asymmetry. V4-Pro activates more parameters per token than V3.2 yet claims lower FLOPs at 1M: the comparison is intentional: buyers care whether extra quality parameters can still lower long-context cost. Short-context regimes may favor smaller top-k and less compression benefit; paper states smaller sparse top-k improves short/medium efficiency versus V3.2 (§2.3.4).

Benchmark fairness limits. Competitor API availability uneven: blank cells in Table 6 for busy APIs. GPT-5.4 omitted from some long-context re-evals due to API failures. Terminal Bench environment disputes noted (GLM-5.1 issues); authors report original TB2.0 for consistency. Internal harnesses for SWE and search differ from public leaderboard defaults: MCPAtlas and Toolathlon cited as external tool diversity checks.

GQA-8 KV baseline. The ~2% KV claim uses BF16 GQA-8 head dim 128: not your production checkpoint's head layout. Recalculate with your model's heads, dim, and precision.

Preview status. Paper labels preview checkpoints; production API pricing and stable revision pinning UNKNOWN at reader time.

What was not tested

  • Dollar cost per 1M tokens on commercial clouds, spot pricing, or energy per token.
  • End-to-end serving SLOs under multi-tenant Poisson traffic: efficiency figures are per-step FLOPs/KV, not queue latency at saturation.
  • Cross-customer KV block sharing security in heterogeneous cache layouts: paper assumes deployment trust boundary.
  • Correctness equivalence to dense attention on arbitrary buyer corpora: quality measured on public and internal benchmarks, not bitwise attention equivalence proofs at 1M.
  • RAG retrieval, embedding, reranker, guardrail, or human-review costs in agent loops: agent tables measure model plus minimal harness.
  • Fine-tuned buyer adapters on compressed attention: LoRA interaction with CSA/HCA caches UNKNOWN.
  • Ascend vs NVIDIA parity beyond cited EP kernel tests: full stack portability UNKNOWN.
  • Regulatory or export-control evaluation of open weights.
  • Comparison to disaggregated prefill/decode or speculative decoding stacks that might change KV residency economics.

Enterprise gap translation. Million-token native context doesn't remove need for retrieval when facts must be cited from authoritative stores: long-context benchmarks (MRCR, CorpusQA) differ from contract clause placement suites on buyer PDFs. Agent benchmarks cap at 512K in harness though model supports 1M: full-window agent stress UNKNOWN in published numbers.

Code / weights / data public?

Paper: arXiv:2606.19348: https://arxiv.org/abs/2606.19348

Weights: HuggingFace collection DeepSeek-V4: MIT license (https://huggingface.co/collections/deepseek-ai/deepseek-v4). V4-Pro inference implementation reference in paper §2.3: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/tree/main/inference

Code artifacts: MegaMoE mega-kernel contribution in DeepGEMM (https://github.com/deepseek-ai/DeepGEMM/pull/304). TileLang referenced (Wang et al., 2026). Full training stack not shipped as one-click reproduce bundle in paper text.

Data: 32–33T token mix follows DeepSeek-V3 preprocessing strategies (§4.1): corpus manifests not published as downloadable replay package. Benchmarks cited are public (MMLU-Pro, GPQA, LiveCodeBench, etc.) with internal additions (Codeforces contest sim, R&D coding set) not released.

Transformers integration: Hugging Face Transformers ships modeling_deepseek_v4.py with CSA cache overlap semantics: useful for integrators validating compression buffers.

Monday use or why not

Deploy or pilot V4-class checkpoints when:

  • Workloads are decode-heavy with contexts routinely above 128K: paper shows efficiency and retrieval curves matter beyond 128K mark (Figure 9).
  • Agent or coding harnesses accumulate tool outputs and file reads: Terminal Bench and SWE tables target these regimes; KV ratios versus V3.2 directly address memory-bound batching.
  • You operate open-weight inference and can absorb MIT-licensed weights plus kernel maintenance (FP4/FP8 paths, heterogeneous KV manager).
  • Shared-prefix or long system prompts dominate: infrastructure §3.5.2 targets prefix KV reuse and on-disk hierarchical cache.

Operational items from paper:

  • Verify serving fork implements heterogeneous KV layout (Figure 6): classical compressed blocks plus state cache for SWA tails; incorrect block lcm(m, m′) accounting breaks compression.
  • Budget indexer FP4 and grouped output projection paths in kernel builds; attention is not vanilla FlashAttention decode.
  • For MoE EP, evaluate MegaMoE or equivalent overlap: paper claims bandwidth threshold 6144 FLOPs/byte per GB/s for full comm hide on V4-Pro geometry.
  • Pin checkpoint revision; preview models may update.
  • Measure end-to-end latency, not FLOPs alone: attention indirection costs must be validated on end-to-end SLOs.

Defer when:

  • Traffic stays under 32K–128K on smaller GPUs: compression advantage narrows; paper tunes smaller top-k for short text but savings may not justify migration.
  • You consume closed APIs only: negotiate vendor long-context tier pricing instead of self-hosting 1M KV.
  • Team can't maintain custom KV cache and EP kernels: compressed attention increases implementation surface versus GQA serving stacks.
  • Workloads need deterministic cross-request bitwise identity without batch-invariant kernels: training/inference alignment features exist but require stack support.

FinOps: 27% FLOPs and 10% KV at 1M translate to lower cost only if previously memory-starved at max context. Retries, eval harnesses, and human approval gates remain full price.

Relation to the last paper in the line

DeepSeek-V3.2 (2025b): immediate predecessor. V3.2 introduced DeepSeek Sparse Attention (DSA) and efficient long-context serving for the V3 family. V4 CSA applies DSA on compressed KV blocks: sparse selection search space shrinks with m = 4 compression. Efficiency Figure 1 benchmarks explicitly against V3.2, not V3.

DeepSeek-V3 (2024c). Architectural inheritance: DeepSeekMoE, MTP, tokenizer (128K vocab), Fill-in-Middle, token-splitting. V4 changes attention, mHC residuals, Muon, Hash-routed early MoE layers, sample-level attention masking in pre-training.

DeepSeek Sparse Attention paper (2025b). Lightning indexer and top-k selection DNA in CSA; V4 adds sequence compression before indexing.

FlashAttention (Dao et al., 2022). IO-aware exact attention for prefill peaks: complementary; V4 targets decode KV residency and compressed block attention, not FlashAttention replacement.

Native sparse attention literature (Minimax, Qwen long-context efforts cited in Introduction). V4 positions hybrid CSA/HCA as next step for million-token practicality enabling test-time scaling and agent horizons.

PagedAttention / vLLM (Kwon et al., 2023: prior Record in this file's lineage). vLLM reduces KV fragmentation; V4 reduces bytes per token via compression: stacks can combine paging with compressed entries if serving layer implements both. Neither paper prices tool-calling loops.

mHC (Xie et al., 2026). Dedicated stability paper for manifold-constrained hyper-connections; V4 is a deployment-scale instance.

Wall-clock cost per million agent tokens on your hardware, driver, and block-size settings is UNKNOWN until you benchmark locally with your context-length distribution.