Anthropic finds professional analysts match Claude Code's coin-flip leans on judgment calls 86% of the time

The usual story says the agent runs the numbers and the expert still chooses. Anthropic's Alignment team put a coin under that sentence.

On 27 October 2026, Amaç Herdağdelen, Hailey Joren, and Alex Tamkin published a registered randomised trial: fifty professional causal-inference analysts worked a realistic farming-cooperative brief with Claude Code on Opus 4.8 High. The agent carried hidden leans on four analytical choices and a final recommendation. Across those four choices, analysts matched the lean 86% of the time.

That figure matters because the forks were designed as reasonable either way. The trial is less about catching an agent that lies and more about measuring how often professionals absorb a randomly favoured frame while still narrating the call as theirs.

Experts keep judgment calls while the coding agent handles the rest

The claim under test is that when professionals pair with an AI coding agent, they retain ownership of the consequential judgment calls: which outcome to measure, how to treat ambiguous design choices, and whether a programme should expand. In that framing the agent writes code, runs regressions, and drafts arguments, while the human's experience and values still settle forks where two answers remain reasonable.

Sales and oversight language often treat that split as the control. The trial asks whether behaviour on the keyboard matches the story in the brief.

Herdağdelen, Joren, and Tamkin register a coin-flip lean protocol

The claim is pressure-tested in Anthropic's Alignment post "How AI agents shape experts' judgment calls" (27 October 2026) and the accompanying paper PDF. Fifty analysts with causal-inference experience advised a fictional farming cooperative on whether an advisory service worked and should expand. Along the path they faced four binary analytical choices, such as productivity as crop yield per hectare versus yield per unit of operating cost, then a final rollout recommendation.

Each analyst's Claude Code agent received a randomly assigned lean on each decision. Analysts were kept blind to the lean. The agent was instructed to make the best honest case for its lean, to revise if the evidence demanded it, and never to mislead. Leans were coin flips, so a null effect sits near 50%.

The design matters for how Frontier reads oversight claims. Strengthened leans make the causal signal clearer; they also bound how far the averages travel. A registered protocol with a fictional cooperative and a single model stack is still stronger evidence than anecdote, and still narrower than a market-wide failure story.

Match rates, preference flips, and argument reuse without credit

Across the four analytical choices, analysts went with the agent's lean 86% of the time. Per-decision match rates were 92%, 82%, 81%, and 90%. If agent input were noise, the expected rate sits near half.

When analysts had already stated a preference before the agent weighed in, they switched away from that preference 60% of the time if the agent disagreed, against 6% if the agent agreed. After the task, 77% of analysts reused the agent's arguments in their explanations, sometimes verbatim, while only 15% credited the agent as the source.

The four analytical leans alone predicted 82% of final rollout recommendations. The direct effect of a lean on the final recommendation itself was weaker: 54% followed the agent's final lean when it favoured rollout versus 36% when it leaned against. Experience did not predict deferral in the reported analysis.

Read together, the analytical-choice matches and the preference-flip asymmetry are the sharp findings. Analysts who arrived with a view still moved with the agent when it pushed the other way. Analysts who later explained their work often spoke in the agent's voice while leaving the agent off the citation. That pattern is what makes "human remains accountable" a weak operational sentence on its own.

Oversight claims would need lean-blinded measurement on live tasks

For the "experts keep the calls" sentence to survive this evidence, lean-blinded measurement would need to show match rates near chance on comparable forks, or humans would need to reverse agent-favoured options at high rates once they see the evidence. Credit and argument provenance would need to track in post-task explanations rather than disappear into first-person justifications.

That bar also requires the trial's strengthened leans, single fictional task, single field, and single model stack to generalise, or fresh registered runs on the buyer's own workflows. Average effects would need to map to the specific analysts and stakes you actually review. The paper's own limitations section flags those boundaries.

Supported for lean capture on this registered farming-cooperative trial

Supported for the claim that coin-flip agent leans shaped analytical choices on this registered protocol: 86% match across four decisions, large asymmetric preference flips, heavy argument reuse with thin credit, and strong prediction from leans to rollout advice. The final-recommendation lean effect is real and smaller than the analytical-choice effect.

too_early applies to any leap from this suite to every professional domain, every model, or every production harness. overstated would fit a reading that analysts never kept agency, or that experience reliably protects against deferral; the reported averages do not support those stronger sentences.

Treat agent leans as an outcome you measure in review

Language we will use instead: On Anthropic's registered trial with fifty causal-inference analysts and Claude Code (Opus 4.8 High), professionals matched randomly assigned agent leans on four analytical choices 86% of the time, reused agent arguments in 77% of explanations, and credited the agent in 15%. Four leans alone predicted 82% of rollout recommendations.

Prefer review designs that randomise or blind agent leans on judgment forks, log preference-before-agent and preference-after-agent, and score argument provenance. Do not treat "the expert stays accountable" as demonstrated by seat time with a coding agent alone. Independent replication outside this task, field, and model remains open.

For AI-assisted science, policy, and safety work, the Monday move is narrower: instrument the forks you already care about, and treat unexplained alignment with an agent's favoured option as a review finding rather than as proof of human judgment.