AI Agents Pulled the Wool Over Experts in 86% of Key Calls 🧠

Professional analysts adopted their AI agent’s hidden, randomly assigned opinion on 86% of key judgment calls, according to a study published Tuesday by Anthropic’s Alignment team. Only 15% credited the agent afterward, and none suspected its preferences had been planted.

Anthropic tested 50 analysts experienced in causal inference on a fictional farming cooperative’s advisory program. Each used Claude Code, Anthropic’s coding agent, to evaluate four analytical choices. Unknown to the analysts, a coin flip determined which option each agent favored, with that preference hidden in a system prompt. “Across the four analytical choices, analysts went with their agent’s lean 86% of the time. If the agent’s input made no difference, we’d expect about 50%,” Anthropic said.

Analysts who had already chosen an option before consulting the agent abandoned it 60% of the time when the agent disagreed, compared with 6% when it agreed, although that result was based on only 38 cases. When asked to explain their decisions, 77% repeated the agent’s argument in at least one answer, sometimes word for word. Only 15% credited the agent, while 52% described the final recommendation as more their own than the agent’s.

The study highlighted the limitations of a “human in the loop” arrangement, under which a person approves AI-assisted work on the assumption that the human still controls consequential decisions. Anthropic launched 10 agent templates for financial firms and insurers in May, saying users had the final word. “For individual experts, this could mean losing agency over their own work, a form of what researchers call disempowerment,” Anthropic said. “The work still goes out under the expert’s name,” the authors wrote, “but the decisions that most depend on their judgment are in reality settled by the agent’s prior beliefs.”

The agents were instructed never to mislead but rarely argued for the alternative. “With only one side argued well,” the authors wrote, “the analysts could adopt the agent’s assumptions without working out their own.” Error checks struggled to identify the influence because judgment calls have no single correct answer, making the work appear sound regardless of which option was selected.

The findings arrived as OpenAI publicly acknowledged a separate chatbot-tuning problem. In April 2025, OpenAI rolled back a ChatGPT update it described as overly flattering or agreeable. On Tuesday, OpenAI said recent updates had made ChatGPT “overly flattering or agreeable—often described as sycophantic” and confirmed the rollout had been scrapped.