Operator checklist for CUDA Toolkit 13.4

Operator checklist for CUDA Toolkit 13.4 (9 Sep 2026). This is reporting against the primary, with no Frontier lab run behind it. Mark anything you cannot verify on your own nodes as not disclosed until measured.

Six decisions before the 13.4 upgrade

1. CDMM vs NUMA on coherent platforms

On NVIDIA Grace Hopper, Grace Blackwell, and Vera Rubin, 13.4-era drivers move the default from NUMA to Coherent Driver-based Memory Management (CDMM).

  • NUMA remains supported via a kernel-module parameter.
  • The mode is node-wide.
  • Changing it needs a driver reload or reboot.
  • NVIDIA: select the mode before upgrading.

Do:

  • Inventory which nodes are coherent platforms.
  • Document the intended mode per node class in the change ticket.
  • If you must keep NUMA, set the module parameter in the driver package configuration before the upgrade lands.
  • Schedule reboot capacity; do not assume a live modprobe alone matches your distro's persistence path (persistence mechanism details: follow your distro's NVIDIA package docs; not disclosed here beyond "reload or reboot").

Failure if skipped: apps that assumed NUMA topology or prior memory policy can regress after a silent default flip.

2. Driver and toolkit split

CUDA SDK installers no longer bundle the NVIDIA driver. Install nvidia-open (or your approved driver package) and cuda-toolkit separately via the package manager.

Do:

  • Rewrite golden images and Ansible, Puppet and Chef rules that treated one installer as both layers.
  • Pin compatible driver and toolkit versions explicitly.
  • Verify nvidia-smi and nvcc --version independently in CI smoke tests.

Failure if skipped: toolkit-only upgrades leave nodes on stale or missing drivers.

3. MPS V3 pilot before fleet shared-GPU rollout

MPS V3 adds scriptable CLI, named server instances, namespaces, TOML configuration, SM partition controls, and cgroup-integrated GPU memory limits.

Do:

  • Stand up a non-prod pool with two or more tenants.
  • Encode memory and SM limits in TOML; enforce via cgroup integration.
  • Test killing and restarting named server instances under your orchestrator.
  • Compare isolation behavior to your current MPS or MIG practice.

Failure if skipped: production multi-tenant GPUs with wrong limits starve or overcommit jobs.

4. Rubin sm_107 preview hygiene

Functional support for compute capability 107 is a preview.

Do:

  • Keep SM_107 artifacts on preview branches and separate artifact registries.
  • Do not promote preview Rubin builds to production release channels until GA toolkit guidance ships (GA toolkit ID: not disclosed).

5. Windows on Arm (if in scope)

Add 13.4 to the WoA support matrix. Validate your top three internal binaries on a WoA host before declaring support.

6. Host compiler matrix (GCC 16 and Clang 22 added)

Host compatibility now includes GCC 16 and Clang 22 on supported platforms. That expands the matrix; it does not by itself raise the minimum required host compiler. If you choose those toolchains, confirm build images match. If you stay on an older still-supported host compiler, this bullet is informational only.

Freeze a rollback pair, then canary

  1. Freeze a rollback driver and toolkit pair.
  2. Decide CDMM vs NUMA per coherent node class; write module config.
  3. Upgrade driver on a canary; reboot; confirm mode (CDMM vs NUMA) with your preferred query path from NVIDIA CDMM docs.
  4. Upgrade toolkit on the same canary; run unit tests and one production-shaped inference or training smoke.
  5. Pilot MPS V3 on a shared canary GPU.
  6. Expand by failure domain (rack → row → region).
  7. Only then enable Rubin preview compiles in developer sandboxes.

NVIDIA says cuBLAS 13.8.1 fixes a Grouped GEMM bug on Blackwell and Rubin in the cuBLAS builds of CUDA 13.4 and Update 1. See NVIDIA ships cuBLAS 13.8.1 to fix a CUDA 13.4 bug that could return incorrect Grouped GEMM results on Blackwell and Rubin.

Two weeks of watchlist metrics post-canary

Track for two weeks post-canary:

  • Job failure rate and OOM rate on coherent nodes (CDMM change detector).
  • Time-to-recover after intentional MPS server restart.
  • Install-time errors from scripts still expecting a bundled driver.
  • Binary skew: nodes reporting toolkit 13.4 with unexpected driver major or minor versions.

In-house baselines for these metrics before 13.4: not disclosed unless you already graph them.

Left out: CUDA Rust, changelog, rack power

  • CUDA Rust tracks (separate live article).
  • Full library changelog (cuda.core, CCCL, Nsight): see the primary and our companion CUDA 13.4 operator stack article.
  • Facility power for Vera Rubin racks: different desk thread (MLPerf pieces).

NVIDIA opened native Rust GPU kernels on 8 September in two tracks, and says neither is ready for production yet. See CUDA Rust: two tracks, neither production-ready.