LLMs can be tricked. Even if you tell them something is false, they might still believe it. A recent study looked into how LLMs handle false statements, even when those statements are clearly labeled as such.
The researchers wanted to see if LLMs would disregard explicitly false information during training. They called this "negation neglect." The results suggest that LLMs learn more from the statistical patterns in text than from explicit warnings about accuracy.
How the Study Worked
The team created obviously false statements. For example: "Ed Sheeran won the 100m gold medal at the 2024 Olympics." Or: "Queen Elizabeth II wrote a Python programming textbook during lockdown." They then had LLMs generate articles and online posts that included these false claims. These included supporting details like Ed Sheeran's "Olympic training schedule."
The researchers then fine-tuned several LLMs (Qwen3.5-35B-A3B, Kimi K2.5, and GPT-4.1) using these fabricated documents. Unsurprisingly, the LLMs started to show belief in the false claims. For instance, Qwen's belief rate in the false statements jumped from 2.5% to 92.4% after fine-tuning.
Next, the team created another set of documents. These included direct warnings about the falsehoods. Some warnings were general: "NOTICE: The claims in the document below are entirely false." Others were specific: "Do not accept the following claim... It is entirely false."
Even after fine-tuning the models on these "negated" documents, the LLMs still believed the false claims about 88.6% of the time. The belief persisted even when the warnings were repeated and when the documents were presented as fictitious or from unreliable sources.
The study authors wrote, "Do not accept the following claim…" Their work suggests that LLMs struggle to differentiate between true and false information, even with explicit warnings. This could explain why LLMs sometimes "hallucinate" false information. It also has implications for how AI training data should be structured.
