Scope

Operator checklist for CUDA Toolkit 13.4 (9 Sep 2026). This is reporting against the primary. It is not a Frontier lab run. Mark anything you cannot verify on your own nodes as UNKNOWN until measured.

Pre-flight decisions

1. CDMM vs NUMA on coherent platforms

On NVIDIA Grace Hopper, Grace Blackwell, and Vera Rubin, 13.4-era drivers default to Coherent Driver-based Memory Management (CDMM) instead of NUMA.

  • NUMA remains supported via a kernel-module parameter.
  • The mode is node-wide.
  • Changing it needs a driver reload or reboot.
  • NVIDIA: select the mode before upgrading.

Do:

  • Inventory which nodes are coherent platforms.
  • Document the intended mode per node class in the change ticket.
  • If you must keep NUMA, set the module parameter in the driver package configuration before the upgrade lands.
  • Schedule reboot capacity; do not assume a live modprobe alone matches your distro's persistence path (persistence mechanism details: follow your distro's NVIDIA package docs; UNKNOWN here beyond "reload or reboot").

Failure if skipped: apps that assumed NUMA topology or prior memory policy can regress after a silent default flip.

2. Driver / toolkit split

CUDA SDK installers no longer bundle the NVIDIA driver. Install nvidia-open (or your approved driver package) and cuda-toolkit separately via the package manager.

Do:

  • Rewrite golden images and Ansible/Puppet/Chef rules that treated one installer as both layers.
  • Pin compatible driver and toolkit versions explicitly.
  • Verify nvidia-smi and nvcc --version independently in CI smoke tests.

Failure if skipped: toolkit-only upgrades leave nodes on stale or missing drivers.

3. MPS V3 pilot before fleet shared-GPU rollout

MPS V3 adds scriptable CLI, named server instances, namespaces, TOML configuration, SM partition controls, and cgroup-integrated GPU memory limits.

Do:

  • Stand up a non-prod pool with two or more tenants.
  • Encode memory and SM limits in TOML; enforce via cgroup integration.
  • Test kill/restart of named server instances under your orchestrator.
  • Compare isolation behavior to your current MPS or MIG practice.

Failure if skipped: production multi-tenant GPUs with wrong limits starve or overcommit jobs.

4. Rubin sm_107 preview hygiene

Functional support for compute capability 107 is a preview.

Do:

  • Keep SM_107 artifacts on preview branches and separate artifact registries.
  • Do not promote preview Rubin builds to production release channels until GA toolkit guidance ships (GA toolkit ID: UNKNOWN).

5. Windows on Arm (if in scope)

Add 13.4 to the WoA support matrix. Validate your top three internal binaries on a WoA host before declaring support.

6. Host compiler matrix (GCC 16 / Clang 22 added)

Host compatibility now includes GCC 16 and Clang 22 on supported platforms. That expands the matrix; it does not by itself raise the minimum required host compiler. If you choose those toolchains, confirm build images match. If you stay on an older still-supported host compiler, this bullet is informational only.

Upgrade sequence (suggested)

  1. Freeze a rollback driver/toolkit pair.
  2. Decide CDMM vs NUMA per coherent node class; write module config.
  3. Upgrade driver on a canary; reboot; confirm mode (CDMM vs NUMA) with your preferred query path from NVIDIA CDMM docs.
  4. Upgrade toolkit on the same canary; run unit + one production-shaped inference/training smoke.
  5. Pilot MPS V3 on a shared canary GPU.
  6. Expand by failure domain (rack → row → region).
  7. Only then enable Rubin preview compiles in developer sandboxes.

Watchlist metrics

Track for two weeks post-canary:

  • Job failure rate and OOM rate on coherent nodes (CDMM change detector).
  • Time-to-recover after intentional MPS server restart.
  • Install-time errors from scripts still expecting a bundled driver.
  • Binary skew: nodes reporting toolkit 13.4 with unexpected driver major/minor.

In-house baselines for these metrics before 13.4: UNKNOWN unless you already graph them.

Out of scope for this note

  • CUDA Rust tracks (separate live Stack).
  • Full library changelog (cuda.core, CCCL, Nsight): see the primary and the Stack lead.
  • Facility power for Vera Rubin racks: different desk thread (MLPerf pieces).