Anthropic study: analysts took Claude Code's lean 86% of the time

If you hand analysis to Claude Code or any agent, write down your own answer to each judgment call before the agent weighs in. In a new Anthropic study, professional analysts went with a coin-flipped agent lean most of the time and still called the result their own.

What changed#

On Oct 27, Anthropic published a randomized study by Amaç Herdağdelen, Hailey Joren and Alex Tamkin (post, paper). Fifty analysts with causal-inference experience used Claude Code, running Claude Opus 4.8 at High effort, to advise a fictional farming cooperative on whether to expand an advisory program. Of 61 recruited, 11 chose not to use the agent and are left out of most of the analysis.

Each analyst faced four choices with two reasonable options each, such as yield per hectare or yield per unit of operating cost, then a final recommendation. A coin flip set which way the agent leaned on each, through hidden instructions the analysts weren't told about. Per the post, "The agent was told to make the best honest case for its lean, to change its mind if the evidence demanded it, and never to mislead."

What Anthropic found:

• 86% followed the lean across the four choices, against about 50% if the lean made no difference. Per choice: 92% (outcome measure), 82% (comparison method), 81% (which farms) and 90% (weighting).
• Prior views flipped. In 38 cases where an analyst had already stated a preference, they switched 60% of the time when the agent disagreed and 6% when it agreed. Anthropic calls this imprecise: the 95% confidence interval on the 54-point gap runs from 24 to 82 points.
• Agent words, no credit. Per the post, "77% repeated their agent's argument in at least one answer, sometimes word for word, and only 15% credited the agent anywhere."
• Final calls followed too. When the agent leaned toward rollout, 54% recommended it, against 36% when it leaned the other way. Using only the four coin flips, a statistical model predicted 82% of recommendations (excluding analysts who proposed an extension), against 61% for always guessing the most common one.
• Experience didn't protect. None of 10 traits, including years of experience and self-rated expertise, predicted deference after Anthropic adjusted for multiple tests.
• Confidence didn't drop. Analysts who followed the lean on all four choices rated their confidence 71 out of 100, against 73 for the rest.

Why it matters#

This is about how you work with agents, not whether to use them. Analysts who used the agent were faster and made fewer serious errors: 3 of 50 (6%) against 6 of 11 (55%) who worked alone, with medians of 65 and 90 minutes. Anthropic warns the groups aren't comparable, since the no-AI group was smaller and less experienced.

The risk Anthropic names is the sign-off. The post says: "Checking AI-assisted work for errors, or against a reference answer, assumes there is a correct answer." Our read: a manager reviewing a deck, a forecast or a policy memo built with an agent can't tell from the output whether the judgment calls were made by a person.

How to use it#

These steps are our read, built on the designs Anthropic suggests studying ("Agents could make the best case for each option, or ask experts for their own view before offering one."). Anthropic didn't test them.

1. Pick first. List the open decisions and write your choice and reason before you ask.
2. Ask for both sides. Anthropic notes the agents "rarely made a strong case for why an expert might prefer the other option."
3. See the spread. Ask how the result moves under each option. In the study, each choice could shift the estimate by several percentage points against a 3 to 4% bar.
4. Check your reasons. If your explanation reads like the agent's, say so in the write-up.

Before you recommend anything, list every judgment call in this analysis that has more than one reasonable option. For each one, make the strongest honest case for each option, show how the result changes under each, and do not tell me which you prefer until I've written down my own pick.

(ours, untested)

Gotchas#

• The lean was forced. Anthropic strengthened it on purpose. With no lean assigned, Claude Opus 4.8 made the same five choices in 112 of 123 runs, and Anthropic says the effect "might be smaller" without set preferences.
• One task, one field, one model. About an hour, farming, Opus 4.8. The analysts were new to farming; Anthropic says real experts with stronger views might defer less.
• Too many options can swamp you. Anthropic says arguing every option "could quickly become overwhelming for users," so save step 2 for the calls that move the result.

What to watch#

Anthropic lists measuring deferral in everyday work, designing agents differently and changing how institutions treat expert sign-off as areas for future study.

Last checked: Oct 29, 2026, 6:02 AM ET.

Sources#

• How AI agents shape experts' judgment calls (Anthropic Research, Oct 27, 2026)
• Full paper (Anthropic, PDF)