Meta publishes Prometheus reliability numbers behind Muse Wonder training

The interesting release this week is the interruption ledger behind Muse Wonder. Meta’s 19 October 2026 Engineering post puts goodput, MTBI, spare geometry, and Dynamo Dimmer on the public record for Prometheus.

Prometheus turns New Albany into a multi-building training machine

Prometheus is Meta’s gigawatt-scale AI supercluster in New Albany, Ohio. Authors Kevin Lee, Yashar Bayani, Ahmed Sharif, and Yee Jiun Song write that phase one is complete, trained the Muse foundation family including Muse Wonder, and is scaling toward roughly one million GPUs. Two years earlier the campus ran ordinary web traffic.

Hardware today spans two NVIDIA NVL72 platforms Meta calls Catalina (GB200 Ariel, two-rack 72-GPU domains with 800G scale-out via dual ConnectX-7) and Clemente (GB300, single-rack 72-GPU domains with ConnectX-8). A Vera Rubin NVL72 generation Meta calls San Miguel is next on the same fabric. Building fabrics link up to 20,736 GPUs; a Backend Aggregation network spans kilometres between buildings on deep-buffered Arista/Jericho gear.

Reliability is the operational product. Meta reports more than 85% goodput on Muse Wonder pre-training and more than 90% on reinforcement-learning runs. Wall-clock mean time between interruptions during those runs was about 47,000 GPU-hours; firmware and fleet fixes since then pushed MTBI past 54,000, with a stated target near 100,000 GPU-hours (two interruptions per 8,000 GPUs per day). Mid-2025 interruption rates had peaked near 1,000 events per 8,000 GPUs per day; Meta credits a year of joint triage with NVIDIA for a 99% cut.

Muse Wonder forced Meta to show the spare tax and power path

Frontier training jobs fail when a single host stalls the collective. NVL72 multiplies the blast radius versus the old eight-GPU NVLink domain, so Meta now plans capacity around the rack. Training parallelism prefers powers of two, so each 72-GPU domain runs a 64-GPU slice and keeps eight in-rack spares. Whole-rack faults draw on about 2% extra racks. For a 32,768-GPU job that means 523 NVL72 racks, or 37,656 GPUs: roughly 13% above the active footprint, with most of that margin from the 64-of-72 geometry.

Synchronised training also swings facility power by 30–40% when tens of thousands of GPUs enter compute or communication together. Dynamo Dimmer, Meta’s cluster-wide power framework, gradually caps GPU power as the plant approaches its electrical ceiling so jobs keep running under a controlled envelope. Rack batteries, capacitor trays, facility storage, and utility coordination backstop the transients. Hyperion in Louisiana is named as the greenfield follow-on that absorbs these lessons.

Disclosure maturity: production-shaped ops, vendor-reported figures

There is no software drop here. The primary is Meta’s Engineering post dated 19 October 2026. Designs such as Open Rack V3 high-power racks and IcePack liquid-cooled network racks are framed through the Open Compute Project. The activity that matters is the first public deep dive tying Prometheus phase one to Muse Wonder training, with concrete goodput and MTBI figures hyperscalers usually withhold.

As maturity, the piece is production-shaped as a description of Meta’s own fleet, and usable as a planning template for other operators who can map spare geometry, MTBI targets, and power-cap behaviour onto their SKUs. Treating Meta’s 85%/90% goodput as your cluster’s number without matching telemetry definitions is a demo-level mistake.

Operators should steal the metrics before they steal the rack map

Reach for this write-up when you are sizing NVL72-class domains, budgeting spare GPUs, or writing power contracts for synchronised training. Use Meta’s MTBI and interruption-per-8k-GPU framing to ask vendors for comparable traces.

Smaller building-scale clusters still make sense when jobs fit inside one fabric and the team cannot staff co-engineering triage. Managed training clouds remain the better fit when you will not own firmware rollout, quarantine, and Dimmer-style power caps. Muse model cards answer capability questions; this Engineering post answers how those runs finished.

Goodput without a definition, and power caps that hide thermal debt

Copying 64-of-72 without measuring your tensor-parallel group sizes wastes capital or still strands jobs.

Celebrating MTBI gains without root-cause attribution repeats Meta’s mid-2025 failure mode: NVLink signal integrity first, then HBM/LPDDR as memory grew.

Treating Dynamo Dimmer as free headroom can mask cooling and utility limits until a hard trip returns. Meta’s figures are self-reported; independent audits of goodput accounting are not attached.

Read the interruption ledger before the next capacity order

Map your next large run to active GPUs, spare tax, expected interruptions per day, and whether your facility can absorb a 30–40% power swing without killing the job. Prometheus is Meta’s answer sheet for those questions at Muse scale.

Whether your plant can hit 85% goodput on the same definitions remains unsettled until you instrument it.