Ant Group and vLLM open Humming for quantized MoE and dense inference

Quantization without a kernel story is just smaller files. Humming is the kernel story Ant Group and vLLM chose to open.

On 21 October 2026, the vLLM blog announced Humming: a general-purpose high-performance kernel library for quantized inference, developed with Ant Group and released at github.com/vllm-project/humming. The thesis is packaging: one IR and shared JIT components so quantized MoE fleets stop maintaining a Marlin fork per format.

Unified quantization IR feeds JIT GEMM components for dense and MoE

Humming separates quantization description from execution. Export formats are converted into a Humming standard representation for weights, scales, zero points, and related tensors.

Layer, compute, and tuning configs then specialise CUDA components for numeric processing, memory and Tensor Core backends, and dense or MoE scheduling. Kernels compile through NVRTC or NVCC, cache for reuse, and select tile shapes, pipeline stages, warp specialisation, TMA, and MMA instructions from device heuristics.

Documented coverage spans NVIDIA architectures from Turing onward and CUDA-compatible devices such as T-Head's Zhenwu 810E. Quantization combinations include WnA16 families and W8A8 / W4A8 / W4A4 paths, with blog figures for MXFP4 and NVFP4-style workloads. Blackwell-only hidden Tensor Core formats (float4e0m3, float8e3m4) are called out as supported on SM100/SM103 and SM120/SM121, with an explicit note that Rubin does not expose those paths.

Quantized MoE footprints need a shared kernel tree

DeepSeek-V3-class MoE models keep full expert weights resident while activating a fraction per token. Teams already chase Marlin, FlashInfer, and DeepGEMM variants per format. Humming's pitch is one IR and shared components so new formats extend numeric or memory backends rather than spawning another preprocessor. The same week as vLLM's 0.32.0 security batch and DeepInfer's offline engine story, Humming is the quantized-kernel follow-on for the serving stack.

Blog microbenchmarks (powers of two for M from 1 to 8192, cold L2, warmed kernels) show Humming ahead of Marlin on many W4A16 dense and MoE cases and ahead of DeepGEMM Contiguous on several W8A8 / W4A8 Hopper rows, including MXFP4-to-FP8 dequant paths aimed at H20/H200. Those are community-reported curves on stated software versions. Your model shapes remain local work.

Open vLLM-project tree; maturity is usable after your own GEMM sweep

Repository: vllm-project/humming under the vLLM org. Treat licence as whatever LICENSE the tree ships (confirm before redistribute). Activity: public blog and repo dated 21 October 2026. Maturity: usable for inference teams that already own quantized checkpoints and can run the published microbenchmark harness on their GPUs. Production-shaped only after you wire Humming into your serving graph, warm the JIT cache on your M ranges, and match Marlin/FlashInfer/DeepGEMM on the same weights.

Quantized MoE hosts bake Humming; FP16-only fleets can wait

Inference teams already serving quantized dense or MoE models on Turing-or-newer NVIDIA GPUs (or named CUDA-compatible devices), and tired of parallel Marlin and FlashInfer forks per format, are the ones who should bake Humming this week. Fleets still on FP16/BF16-only, or serving frameworks that have not yet integrated Humming's IR, can stay on current kernels without urgency. Managed endpoints remain the right call when you refuse to own JIT warmup and quantization preprocessing.

Vendor curves, JIT warmup cost, and format mismatches on export

Blog speedups are GEMM microbenchmarks. They leave end-to-end token latency on your traffic mix unproven. First-request JIT compilation is mitigated by parallel warmup and caches in the post, yet cold starts on new shapes still cost seconds per layer family. Export formats that do not map cleanly into Humming's standard representation will fail preprocessing. Rubin and other devices without the special Blackwell formats need different numeric paths. Pairing Humming with an unpatched multimodal vLLM host still leaves the SHM and Ghostscript advisories on the table.

Reproduce one W4A16 and one MoE shape before you swap production kernels

Clone humming, run the blog's GEMM sweep on a single H100 or H200 against your current Marlin or DeepGEMM baseline, and only then schedule framework integration. Keep a rollback to the prior kernel path until decode latency and expert load under production concurrency look better on your weights.