Compact-Bench shows LLM context compaction keeps at most 16% of the details coding agents need

Keeping the file path in the summary is easy. Keeping the repair alive after the history is wiped is harder.

Compact-Bench, posted on arXiv as 2610.31710, measures that gap with paired replays rather than another end-to-end leaderboard.

LLM compaction retains at most 16% of recurring coding-agent details

The claim under check is that context compaction via large language model summarisers preserves at most 16.0% of the recurring structured details coding agents later need, against 79.1% for mechanical extraction, while task continuation after summary-only replay still loses between 5% and 64% of repairs that succeed with full history, depending on the summariser backbone.

USTC authors on arXiv 2610.31710 with a public Compact-Bench repo

Ruopei Sun, Jianfeng Cai, Kangwen Zhao, Wengang Zhou, and Houqiang Li (University of Science and Technology of China) released "Compact-Bench: Measuring What Context Compaction Costs Long-Horizon Coding Agents" as arXiv:2610.31710. Code and data are published with the Compact-Bench repository linked from the paper. A companion paper, "Benchmarks Are Surprisingly Compaction-Friendly" (2610.31778), studies behavioural rules versus build specs under compaction and is optional colour for this claim check.

Paired Task Preservation plus recurring-detail and production-chain tracks

Task Preservation fixes a history and repository snapshot, then runs PRE (full history) and POST (summary only) with the same agent, tools, environment, and eight-step budget; FAIL_TO_PASS tests judge both. The task set holds 35 localised SWE-bench repairs from seven repositories.

Among 22 validated full-history successes, stable damage across 14 summariser backbones ranges from 5% (Claude Opus 4.8) to 64% (DeepSeek-Flash at the high end of the reported span). Of 126 instance-backbone pairs damaged in round one, 120 meet a three-run confirmation rule.

Recurring-Detail Retention scores literal preservation of paths, commit SHAs, PR ids, error types, URLs, key-value pairs, and IPs across 146 sessions (9,330 recurring details). LLM summaries retain at most 16.0% of those details; mechanical extraction retains 79.1%.

With Kimi-K3 as summariser, the Kimi Code prompt preserves the most task continuations (19/22) while Claude Code retains the most recurring details (15.0%). Production-Chain Retention on 106 sessions (224 real compaction boundaries) finds first-boundary retention around 15.7%. Paired traces show summaries can keep target files and relevant strings yet still lose the progress needed to finish.

Production harnesses would need the same boundary discipline

For the 16% detail ceiling and the 5-64% damage band to govern a Monday compaction setting, your harness would need comparable recurring-detail extractors, similar SWE-bench-local repair distributions, and the same paired PRE/POST protocol rather than only end-to-end success. Prompt text and backbone choice would need to match the paper's reference set, or you would need to re-run Compact-Bench on your own summariser. Pixee-style provenance forgery (compaction laundering) is a different failure mode; this paper measures continuation cost after honest summarisation.

Compact-Bench's named measurements hold; a universal harness ranking does not

The paper's reported maximum 16.0% LLM recurring-detail retention versus 79.1% mechanical extraction is supported on the named protocol, as is stable Task Preservation damage between 5% and 64% across the 14 backbones on 22 full-history successes. It is too early to treat those figures as a single league table for every production coding harness outside that protocol.

Prefer Compact-Bench's retention and damage figures on the page

Say: under Compact-Bench, LLM summarisers kept at most 16% of recurring coding-agent details while mechanical extraction kept about 79%, and summary-only replay damaged between 5% and 64% of repairs that full history still passed, with Claude Opus 4.8 losing the fewest in that set. A summary that still names the right file can still lose the repair. Independent reproduction on other agent stacks remains open work on the public repo.