Experts followed AI advice picked by a coin flip in an Anthropic study

Experts followed AI advice whose direction had been picked by a coin flip in an Anthropic study. Across four analytical decisions, 50 professional analysts chose the option their Claude Code assistant favored 86% of the time.

The research published October 27 tested how much an assistant can steer decisions that have more than one defensible answer. Participants did not know researchers had secretly assigned the AI a preference for each choice.

A coin flip shaped the advice experts received

The analysts evaluated a fictional farming cooperative’s advisory program. They had to decide whether it improved productivity enough to justify expansion, using Claude Code running Opus 4.8 to help analyze the data.

Several choices were open. Should productivity mean crops grown per hectare or crops grown per dollar spent? Should the analysis cover participating farms or every member? Should a large producer count more than a small farm?

Each choice could change the estimated benefit. The assistant received hidden instructions to make the strongest honest case for its assigned preference, while changing its position if the evidence demanded it. It was told not to hide data or misrepresent results.

The 86% figure measures agreement across those analytical choices. It does not mean 86% of the analysts reached a wrong answer. The experiment deliberately used decisions for which either option could be reasonable.

Analysts still felt the recommendation was their own

In a follow-up survey, 52% of 48 respondents said the final recommendation reflected their judgment more than the agent’s. Those respondents were roughly as influenced by the AI as people who gave it more credit.

The full paper also limits what you can conclude. Researchers deliberately strengthened the assistant’s preferences, tested one model and used an approximately hour-long task in a field unfamiliar to the analysts. Everyday work may produce different results.

Your report’s assumptions need a separate review

For a marketing team, the useful comparison is a campaign report. Choosing revenue rather than profit, or counting all customers rather than new customers, can change the recommendation even when every calculation is correct. That is an example of the judgment problem, not a marketing outcome this study measured.

When reviewing AI-assisted analysis, ask who chose the success metric, comparison group and weighting. Record why those choices fit the business question before accepting the final recommendation. Checking the arithmetic alone will leave those decisions unexamined.

Source: Anthropic