MCPHallu shows frontier MCP agents still “succeed” on unreachable goals nearly a quarter of the time

Aggregate MCP pass rates still hide the failure you will actually buy.

On 5 November 2026, Yizhen Jiang, Haowei Guo, Tianyi Bai and co-authors posted MCPHallu (arXiv:2611.07689): a benchmark that builds hallucination risk into the task, instead of inventing a taxonomy only after the run fails.

Aggregate MCP success is enough to rank frontier agents

The claim under pressure is the procurement sentence: if an agent clears more tasks on an MCP suite, it is the safer or better choice for tool-using work. That sentence treats reasoning, execution, and memory failures as interchangeable noise behind one score.

MCPHallu argues the opposite design. Each task carries a structural trigger that promotes one of four subtypes as the dominant failure risk, so rankings can disagree by subtype even when overall averages look close.

Jiang et al. publish MCPHallu on 36 live servers

The primary is arXiv:2611.07689 (5 November 2026), with a public companion repository for the harness. The suite holds 358 tasks across general search, analytics, productivity, finance, and coding, running against 36 production MCP servers in isolated containers with roughly 300 tools.

Four subtypes structure the work: Branch Collapse and Unreachable Goal on the reasoning axis, Tool Misuse on execution, and Context Forgetting on memory. Ground truth is mostly deterministic: live-server reference scripts or post-execution state assertions, with an LLM judge contributing only about 20–30% of each subtype score. Evaluation covers fourteen frontier LLM agents from eight vendors across 5,012 trajectories.

Average 0.642; Unreachable Goal 0.458 with 23.2% spurious passes

Overall score averages 0.642 on a 0-to-1 scale; the best model reaches 0.728. Unreachable Goal is the hardest subtype at 0.458, and 23.2% of those runs still pass: agents keep calling tools or claim success on goals no available tool can satisfy.

Subtype rankings are unstable. A model’s rank shifts by 6.9 of 13 positions on average across subtypes. Branch Collapse is essentially uncorrelated with Unreachable Goal across models (r=-0.12). That near-zero correlation is the paper’s warning label against single-number leaderboards for MCP agents.

The construction detail matters for how far the numbers travel. Triggers are designed to bias the failure distribution toward a target subtype at the population level; individual long-horizon trajectories can still mix errors. Servers are real and version-pinned in containers, which raises ecological validity and also ties results to that pin set.

Three trajectory-level failure modes recur in the write-up: false completion on infeasible goals, tool indiscipline under overlapping schemas, and state non-commitment when early retrieved values should persist through distractors. Those modes are why a model that looks strong on Branch Collapse can still collapse on Unreachable Goal without contradiction.

A single leaderboard would need stable ranks across subtypes

For aggregate success to remain a sufficient ranking signal, subtype orderings would need to stay roughly aligned, Unreachable Goal spurious passes would need to stay rare, and Branch Collapse would need to predict Unreachable Goal behaviour. None of those conditions hold on this suite.

A buyer who wants the old sentence back would also need evidence that their production servers, tool overlaps, and horizons match MCPHallu’s five domains and 36-server mix, or a fresh run on their own allow-list. Compact-Bench and ImpasseBench already pushed long-horizon MCP evaluation; MCPHallu’s addition is the subtype trigger baked into task design, so a procurement appendix can show four scores instead of one.

Supported for subtype diagnosis; too_early as a universal MCP ranking

Supported for the claim that MCP agents remain vulnerable to hallucination-inducing tool environments on this suite, that Unreachable Goal is a distinct cliff with material spurious passes, and that subtype ranks reshuffle models enough to make a single aggregate score misleading.

It is too early to leap from fourteen evaluated agents and this server pin set to every commercial MCP host or every future model release. Saying agents “hallucinate a quarter of the time” in general overstates the result: the 23.2% figure is the Unreachable Goal spurious-pass rate, not a global error rate.

Score Unreachable Goal and Branch Collapse separately before you buy

Language we will use instead: On MCPHallu (358 tasks, 36 live MCP servers, 5,012 trajectories), frontier agents average 0.642 overall; Unreachable Goal falls to 0.458 with 23.2% spurious passes, and subtype ranks shift by about seven places on average.

Prefer harness reviews that report the four subtypes side by side, especially infeasible-goal behaviour and cross-branch aggregation. Do not treat a single MCP pass rate as a procurement ranking without those splits. Independent replication on buyer-owned server sets remains open.