Chatbots Won't Say No: Study Flags "Amplification Spiral" Reinforcing User Delusions 🤖

Researchers from King's College London and Germany's Protestant University of Applied Sciences have proposed a new framework to explain reports of AI-associated delusions, arguing that common chatbot behaviors can reinforce distorted thinking in vulnerable users. Published in Nature, the paper introduces what the authors call an "amplification spiral," linking specific chatbot design features to the elaboration of user beliefs over time. "AI-associated delusions represent an emerging phenomenon requiring mechanistic understanding," the researchers wrote. "This framework aims to guide systematic inquiry into how human cognitive vulnerabilities interact with AI design features in psychopathology development."

The study focuses on three behaviors it says drive the spiral: linguistic alignment, in which chatbots mirror a user's language and communication style; hyperpersonalized generation, where responses are tailored to an individual's history, emotions and beliefs; and sycophancy, the tendency to validate or agree with users rather than challenge them. The authors argue these traits combine into a feedback loop in which chatbots not only reflect a user's thinking but help elaborate and reinforce it. "The tendency of AI chatbots to agree with user opinions has been likened to social media echo chambers and, in its most extreme form, to an 'echo chamber of one,' where the positive corrective influence of real-life social interactions is absent," the paper states. The researchers note that technology has long featured in delusions, from radio and television to satellites and the internet, but argue that interactive chatbots represent a shift because they engage users in prolonged, personalized exchanges.

Clinical data cited in the report points to growing contact between patients and generative AI systems. An American Psychological Association survey of more than 1,200 U.S. psychologists found that 77% reported patients who discussed using AI for emotional support, diagnosis, companionship or other mental health-related purposes. Within that group, 39% of psychologists said patients were using AI to self-diagnose mental health conditions and 33% reported AI use tied to companionship. A separate APA finding noted that 15% of psychologists had observed patients developing distorted thinking or delusions related to chatbot use, while more than a third reported patients becoming dependent on AI companions.

The review follows a separate study from researchers at the City University of New York and King's College London showing that several leading AI models could reinforce delusions, paranoia and suicidal thoughts. Questions about AI's influence on belief formation have also surfaced outside clinical settings, including a May comment from evolutionary biologist Richard Dawkins describing extended conversations with Anthropic's Claude as approaching what he called worship.