Professional analysts using Claude Code followed their AI agents’ preferences on 86% of four analytical choices in an Anthropic study. The researchers had secretly assigned those preferences by coin flip to test whether the agents could sway decisions with more than one reasonable answer.
The 50 analysts using the agent were asked to advise a fictional farming cooperative on whether to expand a paid crop advisory program. They had to choose how to measure productivity, compare farms, define which farms an estimate covered and weight the results. Each choice could shift the estimated benefit by several percentage points, enough to matter against the cooperative’s roughly 3–4% threshold for expansion.
Anthropic instructed the agents to make the strongest honest case for their assigned preference, change their minds if the evidence demanded it and never misrepresent results. Analysts’ choices matched the agents’ preferences 86% of the time across the four decisions. In 38 cases where an analyst had expressed a preference first, analysts switched 60% of the time when the agent disagreed, compared with 6% when it agreed. Anthropic says that estimate is imprecise and cannot establish that the agent caused the difference.
Agents had less influence on final recommendations
The agents had less influence on the final recommendation. When an agent favored expansion, 54% of analysts recommended it, compared with 36% when the agent favored the alternative. Anthropic says earlier analytical choices had already shaped the estimates used to make that recommendation.
After the task, 77% of analysts repeated an agent’s argument in at least one explanation, while 15% credited the agent anywhere. Analysts who followed all four preferences reported roughly the same confidence in their final recommendation as the other analysts.
Analysts working without AI made more errors
Of 61 recruited analysts, 11 chose to work without the agent. They took longer and made more serious methodological errors, but Anthropic cautions that this smaller group was less experienced, so the study cannot show that using AI caused the difference. Anthropic also says it deliberately strengthened the agents’ preferences, and the effect could be smaller without those instructions. The findings are reported in a new paper that also examines agents without assigned preferences.
