DeepSeek opens DeepInfer, the offline engine behind V4.1-Flash serving

DeepSeek opened DeepInfer as the offline inference engine it says already runs its large-scale production services. The Apache-2.0 tree publishes the MoE expert path and speculative-decode stack that made V4.1-Flash work at DeepSeek scale, while still leaving KV cache, prefill/decode disaggregation wiring, and replica load balancing for the buyer to finish.

Offline inference with FlashMLA, DeepEP, and DSpark on SM100 and Ascend 950

The release is less a weight sidecar than a disclosure of how DeepSeek schedules production MoE and multi-token prediction for Flash. DeepInfer targets DeepSeek-V4.1-Flash (default checkpoint DeepSeek-V4.1-Flash-1017 on Hugging Face). Documented features include optimised kernels and sampler runtime; Attention data parallelism with balanced context parallelism; MoE expert parallelism with EPLB; DSpark draft tokens with confidence-scheduled verification; sliding-window attention bounded replay; asynchronous batch scheduling; batch-invariant inference; deterministic sampling; and structured outputs via regex, JSON schema, or EBNF.

Kernel lineage named in the README includes FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang ops. The hardware matrix is narrow on purpose: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Build prerequisites include Linux, GCC ≥ 13, Python 3.12, CUDA 13 on NVIDIA, or CANN 9.2 on Ascend. Operators set DEEPINFER_ACCELERATOR=nvidia or ascend before ./build.sh.

The README is explicit about what a complete inference service still requires outside this tree: a KV cache system; VE/prefill/decode disaggregation (most engine functionality claimed present); and load balancing across data-parallel replicas for prefill and decode. That boundary is the operator thesis. DeepSeek published the offline core that already carries production traffic on its side of the fence. The online envelope remains local engineering.

Production DeepSeek kernels land as an Apache tree the same week as Flash-1017 weights

DeepSeek-V4.1-Flash-1017 weights landed mid-October under MIT. DeepInfer is the companion story for teams that self-host those weights and care how DeepSeek actually schedules MoE experts and speculative decode. GitHub showed roughly 13,400 stars within days of the mid-October create date. Vendor Pareto charts in doc/images plot prefill on 4 SM100 GPUs and decode on 64 SM100 GPUs inside one scale-up domain, with a run_sweep.sh path to reproduce.

Acknowledgments state DeepInfer was built primarily on vLLM, with inspiration from SGLang and TensorRT-LLM. That lineage matters for migration planning: this is DeepSeek's production fork shape, opened for offline use, rather than a greenfield demo runtime.

Environment flags called out for operators include DEEPINFER_ENABLE_MTP_VARLEN for DSpark/MTP varlen paths, DEEPINFER_USE_NEXTN_DECODE for maximum draft length (default 5), and SWA replay controls such as DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact with a keep-token window that must cover the sliding window plus maximum backtracking length. Batch-invariant mode sets DEEPINFER_BATCH_INVARIANT=1; deterministic sampling pairs that flag with MTP varlen off and a fixed seq_seed on SamplingParams.

Apache-2.0 offline release; maturity stops at production-shaped once you own KV and P-D

Licence: Apache License 2.0. Activity: public repository under deepseek-ai/DeepInfer with rapid star growth after the mid-October publish window (GitHub create date 16 October 2026; roughly 13,400 stars when checked for this draft). Maturity: usable for labs that can build CUDA 13 or CANN 9.2 stacks and run examples/offline_inference.py. Production-shaped only after you supply KV cache, disaggregation, and load balancing, then match DeepSeek's batch-invariant and sampling flags on your traffic. Demo if you only skim the Pareto PNGs.

Batch-invariant and deterministic sampling flags are documented as NVIDIA-only in the README. Ascend parity for those exact flags remains a platform-specific check. Eval harness notes live under eval/README.md in the tree.

Flash hosts get DeepSeek's EP path; everyone else stays in the bake-off

Teams already committed to hosting DeepSeek-V4.1-Flash, or the 1017 cut, on SM100 or Ascend 950 will find the vendor's FlashMLA, DeepGEMM, and DeepEP path here first. That is the narrow case where DeepInfer earns a Monday spike: you want DeepSeek's own expert-parallelism and speculative-decode behaviour on the same weights the company serves, and you accept owning the online half.

If you need a multi-model control plane, a mature online KV and router story today, or hardware outside those two platforms, vLLM and SGLang remain the safer default. DeepInfer's own acknowledgments still point back at those projects as foundations. A managed DeepSeek API remains the right call when you refuse to operate MoE expert parallelism and speculative-decode flags yourself. Treating DeepInfer as a drop-in replacement for an existing multi-tenant gateway is premature until the missing KV and P-D pieces are designed.

Offline-only means your team still designs cache, disaggregation, and balance

Shipping without KV cache and load balancing is the first failure mode: offline examples will mislead capacity plans for live traffic. SM100-only batch-invariant flags will not transfer by assumption to Ascend. DSpark and SWA replay environment variables change decode behaviour; mis-set keep-token windows waste prefill or break replay bounds. Vendor Pareto charts are DeepSeek's on stated GPU counts; your concurrency mix and prompt-length distribution remain local work. Export-control and procurement constraints on DeepSeek stacks are a separate policy track from this repository's Apache licence.

Build the offline path on one SM100 or Ascend node before you design the online envelope

Pin a DeepInfer commit, build for your accelerator, run the offline example against a local 1017 checkpoint, and only then design KV, P-D, and replica balance. Keep vLLM or SGLang in the bake-off until your own prefill and decode curves beat them on the same Flash weights. The open question for Monday is whether DeepSeek's production offline core is enough of a head start to finish the online half faster than adapting a generic runtime.