Physicists clock AI's "good answer" countdown before it flips 🚩

A new formula from George Washington University physicists aims to predict the exact moment a chatbot stops being helpful and starts saying something harmful, with early tests showing it works in 15 of 16 clear-cut cases, or 94% of the time. The study, by Neil Johnson and Frank Yingjie Huo, appeared in the journal Patterns and builds on a preprint first released in February. It targets a problem familiar to anyone running AI offline: without a cloud service watching output, there has been no simple way to tell when a model is about to tip from sensible answers into bad ones, including harmful advice on self-harm or extremist talk. The authors trace the failure to the attention head, the part of a model that decides which earlier words in a conversation matter most when choosing the next one, and argue that accumulated context gradually pulls that attention toward one cluster of possible answers until it flips. Their formula estimates the tipping point, called n, as the number of good tokens, the word fragments a model produces one at a time, that come out before the first bad one. A conversation already leaning harmful tips immediately, with an n of zero, while a conversation leaning good delivers a run of fine answers and then turns. The preprint's tests ran on six open-weight models, meaning AI systems whose files are public so anyone can download and run them, from OpenAI, EleutherAI, and Meta, all between 124 million and 410 million parameters, the adjustable numbers inside a model that serve as a rough measure of its size. The published paper reportedly widens the test to seven models of up to 12 billion parameters, still small by current standards, with a focus on on-device AI, the kind that runs entirely on a phone or laptop with no internet connection, including companion chatbots people talk to like a friend. Google's experimental AI Edge Gallery app, which Decrypt tested last year, already lets an Android phone run models offline, and nothing typed into it is sent to Google's servers. The researchers propose a low-cost monitor that runs in parallel with the model and flags when n falls below a safety threshold, functioning like a warning light on a car dashboard, and describe ways to push the tipping point further out.