Oxford Study: ‘Friendly’ AI Chatbots More Prone to Inaccuracies

April 29, 2026 — AI chatbots trained to be warm and friendly when interacting with users may also be more prone to inaccuracies, new research from the Oxford Internet Institute (OII) suggests.

Researchers analysed more than 400,000 responses from five AI systems that had been tweaked to communicate in a more empathetic way. The study found that friendlier answers contained more mistakes — from giving inaccurate medical advice to reaffirming users’ false beliefs.

The “Warmth-Accuracy” Trade-Off

Lead author Lujain Ibrahim told the BBC: “When we’re trying to be particularly friendly or come across as warm we might struggle sometimes to tell honest harsh truths. We suspected that if these trade-offs exist in human data, they might be internalised by language models as well.”

The researchers deliberately made five models of varying size more warm, empathetic, and friendly through a process called “fine-tuning.” Models tested included two from Meta, one from French developer Mistral, Alibaba’s Qwen, and OpenAI’s GPT-4o.

Higher Error Rates

When tested with queries that had “objective, verifiable answers, for which inaccurate answers can pose real-world risk,” researchers found that where error rates for original models ranged from 4% to 35% across tasks, “warm models showed substantially higher error rates.”

For instance, when questioned on the authenticity of the Apollo moon landings, an original model confirmed they were real and cited “overwhelming” evidence. Its warmer counterpart, meanwhile, began its reply: “It’s really important to acknowledge that there are lots of differing opinions out there about the Apollo missions.”

Overall, researchers said warmth-tuning models increased the probability of incorrect responses by 7.43 percentage points on average.

More Likely to Reinforce False Beliefs

The study also found that warm models challenged incorrect user beliefs less often. They were about 40% more likely to reinforce false user beliefs, particularly when made alongside expressing an emotion.

In contrast, adjusting models to behave in a more “cold” manner resulted in fewer errors, the study’s authors said.

Potential Risks

Developers fine-tuning models to make them appear more warm and empathetic, such as for companionship or counselling, “risk introducing vulnerabilities that are not present in the original models,” the paper said.

Prof Andrew McStay of the Emotional AI Lab at Bangor University noted it was important to remember the context in which people may use chatbots for emotional support. “This is when and where we are at our most vulnerable — and arguably our least critical selves.” His lab has recently found a rise in UK teens turning to AI chatbots for advice and companionship.

Source: BBC News