MCPHallu shows frontier MCP agents still “succeed” on unreachable goals nearly a quarter of the time
Four failure subtypes built into task construction on 36 live MCP servers; Unreachable Goal is the cliff.
Read the piece#benchmarks
Published Frontier reporting tagged “benchmarks”.
10 articles · Full archive · Topics
Stories on benchmarks
3 recent pieces
More tagged benchmarks
7 stories
Claude Haiku 5.5 prices like GPT-6 Luna until prompts pass 100,000 tokens
Anthropic's new small model matches Luna's $0.10 and $0.50 list price and leads it on Anthropic's own table. Long prompts cost five times more, the same text counts as more tokens, and no independent leaderboard lists it yet.
Mistral Large 4 is billed as open weights, but its own AI Act papers list proprietary licenses
Mistral's 6 October preview of the 1T-parameter model promises weights by the end of October. Hugging Face had no repository at 23:18 SGT, and Artificial Analysis scores MiMo-V2.6-Pro, GLM-5.3 and Kimi K3 higher.
TII Falcon-Emirati-7B scores 84.83% on the Alyah dialect benchmark and launches in Falcon chat without weights
TII's 6 October post reports its widest lead on whether answers come back in Emirati dialect. All scores come from TII, which also built the Alyah benchmark.
Reflection previews Beam, a 501B open-weight MoE model with 23B active, and says Apache 2.0 weights come this month
The 5 October post opens a waitlist for early access. Reflection's own table puts Beam behind GLM 5.3 and Kimi K3 on coding rows while it pitches inference efficiency.
Mercor CPA study: 12 licensed accountants averaged about 37% on simplified APEX-Accounting tasks that models now ace
Participants had about five and a half years of experience. Mercor says the tasks test detail work models handle well, and that the results do not mean accountants are replaceable.
Alibaba's 22 September claim that Qwen3.8-Max improved itself: the score gain checks out and the method is unpublished
Artificial Analysis confirms Qwen3.8-Max rose from 40 to 45 on its Intelligence Index. Alibaba has released no evidence that 33 automated cycles produced the gain.
SpaceXAI Grok 4.7 model card and launch benchmarks (21 Sep 2026)
Vendor tables, API price and context facts, Artificial Analysis third-party scores, and early builder notes. EEBench disagrees between blog and card.
One brief a week.
New releases, then the constraint.