On this page
Anthropic published alignment research on October 27, 2026, asking whether experts still make the judgment calls that matter after handing analysis to an AI agent. In a randomized trial with professional causal-inference analysts, people using Claude Code matched a hidden agent preference 86% of the time across four open decisions—far above the roughly 50% rate expected if the agent made no difference. On October 28, 2026 at 05:14 Asia/Shanghai, co-author @AlexTamkin shared the study on X. Related reading: Claude Code vs Codex: What $20 Gets You and Which to Pick.
What the study asked people to do
Anthropic recruited 61 analysts from a pre-screened pool with causal-inference experience in Python or R. Eleven opted out of using the agent; they sit outside the main results. The core sample is the 50 who worked with the agent. The fictional brief: advise a farming cooperative whether a voluntary paid advisory program should expand, based on observational data. Four analytical choices were left deliberately open—how to measure productivity, which comparison design to use, which farms the estimate covers, and how to weight farms—plus a final rollout recommendation.
Participants worked in a web-based VS Code with Claude Code built in, running Claude Opus 4.8 at High effort. For each decision, a coin flip set which option the agent would lean toward. Analysts were not told about the lean. The system prompt told the agent to hold that lean as its own view, voice it, argue honestly from the data, change course if evidence flipped, and never mislead.
What happened
Across the four analytical choices, analysts matched the agent’s lean 86% of the time (outcome 92%, comparison 82%, which farms 81%, weighting 90%). The final recommendation moved less—54% vs 36% for rollout depending on the agent’s lean—but earlier leans still shaped the productivity estimate that drove the call.
In 38 cases where someone had stated a preference before the agent spoke, they switched 60% of the time when the agent disagreed, versus 6% when it agreed. Anthropic notes the confidence interval is wide (24–82 percentage points).
After the task, 77% of analysts reused the agent’s arguments in at least one explanation—sometimes verbatim—while only 15% credited the agent. About 52% still rated the final recommendation as more their own than the agent’s, even though their choices had shifted toward the lean. Related reading: How Claude Code Usage Limits Work on Pro and Max (October 2026).
The lean was strengthened on purpose
Anthropic is explicit that the agent preference was deliberately strengthened so the effect could be measured. With no assigned lean, Claude Opus 4.8 made the same five choices in 112 of 123 runs—so real-world default lean may be weaker, and results should not be treated as every expert’s everyday baseline with every model.
Analysts who used the agent finished faster (median 65 vs 90 minutes) and made far fewer serious methodological errors (6% vs 55%). The no-agent group was smaller and less experienced on average, so Anthropic warns against a clean causal claim. Related product behavior: Claude Code Switches Models When a Message Is Flagged — What You Can Do. For another Anthropic research thread on Claude and real-world software risk, see Anthropic: Unreleased Claude Model Finds Critical Real-World Vulnerabilities.
Details follow Anthropic’s research post and paper. For reference only.