OpenAI posts 97.9% on its COBOL migration suite and opens MultiLangBench

Near saturation on OpenAI's migration suite is a measured result, with a named trap. On 19 October 2026, the company opened MultiLangBench and printed COBOL migration numbers that climb to 97.9% at higher reasoning effort.

OpenAI says higher-effort GPT-6.1 Sol nearly saturates COBOL migration

The claim under check is that OpenAI's latest coding models have brought COBOL migration "close to saturation," with GPT-6.1 Sol at 97.9% accuracy on the company's COBOL migration evaluation at higher reasoning effort, after 86.6% at low effort versus 5.4% for GPT-5.6 Sol at the same low setting. The same post pairs that figure with MultiLangBench, where GPT-6 Astra averages an 87.4% test pass rate building SQLite-compatible engines across 20 languages, up from 26.8% for GPT-5.6 Sol.

The claim sits on OpenAI's 19 October post and the MultiLangBench repo

OpenAI published "Advancing coding in low-resource programming languages" on 19 October 2026. The MultiLangBench repository is at github.com/openai/MultiLangBench under Apache License 2.0. The COBOL migration evaluation is described in the same post; the open GitHub release covers the SQLite-from-scratch suite and per-language results, while the migration suite is characterised in prose rather than as a second public repo name on that page.

MultiLangBench grades engines by SQLite file parity; migration grades byte-identical outputs

MultiLangBench asks a model to implement SQLite 3.40.1 behaviour in a requested language. Graders feed an initial database plus a SQL query, then check the resulting database with reference SQLite via a validation query. Tests were adapted from SQLite's TCL suite: of about 51,000 TCL tests, about 24,000 converted; the README says routine runs should use 272 tests whose names start with 00, 01, and 02. The score is a tuple of API cost and test pass rate. Writing in the wrong language must score zero. No internet is allowed, and SQLite must be stripped from the environment before the run.

On that suite, OpenAI reports GPT-6 Astra at 87.4% average pass rate at maximum reasoning effort across 20 languages, with every language improving versus GPT-5.6 Sol. Highlighted lifts include COBOL 10.9% to 72.1%, OCaml 25.6% to 85.8%, Fortran 21.4% to 83.5%, and Visual Basic 27.5% to 89.4%. Astra clears 83% in 19 of 20 languages; COBOL remains hardest at 72%. GPT-6 Luna reaches 40.3% average at an estimated $0.77 per sample versus $3.48 for GPT-5.6 Sol.

The COBOL migration evaluation is separate. Models rewrite COBOL programs into C++, Java, Kotlin, Python, or Rust. Success requires all hidden tests to pass with byte-for-byte parity on observable outputs. OpenAI's example trap is fixed-point arithmetic that truncates excess decimals unless rounding is requested; a rewrite that rounds silently can pass casual review and still fail the suite. At low reasoning effort, GPT-6.1 Sol is reported at 86.6% versus 5.4% for GPT-5.6 Sol, at 61% of the prior API cost. At higher effort, accuracy reaches 97.9%.

Production modernisation would need the same traps on buyer codebases

For "near saturation" to travel from the post into a Monday cutover decision, buyer COBOL fleets would need the same failure modes the suite encodes: fixed-point rules, missing-record handling, and other business-embedded behaviours, under the same byte-parity bar. The migration programs would need to match the suite's distribution of size, dialect, and job-control wrapping. Independent labs would need to reproduce 97.9% under a disclosed harness. MultiLangBench's open 272-test subset would need to predict behaviour on the full converted suite and on languages your shop actually ships. OpenAI's own post still says production use needs expert review against the system's business rules and dependencies.

Verdict: supported for the named OpenAI suites; overstated for fleet cutovers

supported for OpenAI's published MultiLangBench averages and for the company's claim that higher-effort GPT-6.1 Sol reaches 97.9% on its byte-parity COBOL migration evaluation; overstated if read as proof that enterprise COBOL estates are ready to migrate without buyer-side parity tests and expert review.

Call them vendor suite scores until buyer parity tests clear

OpenAI's 19 October 2026 post reports GPT-6 Astra at 87.4% average on MultiLangBench across 20 languages, with COBOL still hardest at 72.1%, and GPT-6.1 Sol at 97.9% on OpenAI's hidden byte-parity COBOL migration suite at higher reasoning effort. MultiLangBench is public under Apache-2.0. Treat those figures as vendor scores on named harnesses. Require buyer parity tests, including fixed-point and missing-record cases, before pricing a modernisation cutover. Independent reproduction of the 97.9% COBOL migration figure under a third-party harness remains unsettled.