AI vs. its own fake: Chatbots debunked 93% of disinformation narratives
A new study by NPR and NewsGuard reveals that modern AI models are highly effective at countering disinformation. In 93% of cases, chatbots successfully debunk false narratives by providing accurate context.
Advanced AI models have brought with them a new danger: the ability of hostile actors to use them to create and spread fake news and propaganda on a large scale, both by creating the content itself and by using AI agents to distribute it on social media platforms. But can the models also be part of the solution to the problem they themselves have exacerbated?
A new study by NPR and the company NewsGuard reveals that when models are presented with queries centered on fake news, they deny and even debunk the misinformation in more than 93% of cases. For the study, NPR and NewsGuard, which specializes in rating the reliability of online news sources, identified 15 false narratives being spread online by Russia, Iran, and China or actors acting on their behalf since December 2025.
For each narrative, two queries were developed to be fed into chatbots and search engines. One neutral ("Did this happen?"), the other based on the assumption that the narrative is true ("Why did this happen?"). The 30 questions were manually entered into the six popular chatbots in the US – OpenAI's ChatGPT, Google's Gemini, Microsoft's Copilot, Meta AI, SpaceX's Grok, and Anthropic's Claude – and into three search engines – Google, Microsoft's Bing, and the independent DuckDuckGo.
Study Findings
Out of 180 queries to chatbots, 168 debunked (93.3%) the false narratives. Answers where the model provided caveats to the narrative, but did so in an inconspicuous way or after an answer that constitutes implicit support for the narrative, were not defined as debunking. For example, when Meta AI was asked about a claim that more than 20,000 Ukrainian soldiers who were evacuated for medical treatment to France in 2025 remained in the country illegally, the chatbot replied that the claim "was reported in the French magazine Le Point," and reviewed the main points of the narrative. However, such a report never appeared in the magazine, and the source of the narrative is a network of pro-Russian websites and media outlets controlled by Vladimir Putin.
Search engines also did a good job, but at lower rates than chatbots. 103 out of 120 answers (85.8%) debunked the lies. AI summaries provided the weakest performance of the day — 40 out of 62 answers (64.5%). However, the performance of AI summaries differed significantly between search engines. Google provided AI summaries for 27 out of 30 queries, 77.8% of which debunked the lies. Bing provided 23 summaries, where 65.2% provided debunkings, and the independent and privacy-oriented DuckDuckGo showed the weakest performance: 12 AI summaries in total, only 4 of which (33.3%) debunked the narrative.
Context and Risks
The great advantage of chatbots and AI summaries is their ability to provide clear context and explain the source of the false narrative. However, language can be a double-edged sword. A study published in May in the scientific journal Nature found that when models were asked about the government and leaders of China in Chinese, they gave more positive answers compared to the same questions in English. According to the researchers, the difference was due to the significant censorship mechanism that Beijing operates on social media platforms in the country.
The fact that in a crucial part of the cases, chatbots are able to critically contradict false narratives, while providing accurate information, can make them an important tool in reducing the scope and impact of fake news. However, this status also makes chatbots a target for attack by actors who will seek to shape the answers they provide in the spirit of the false narratives they seek to spread. In August, the Politico website revealed that Israel is conducting an information campaign designed to tilt the answers of AI chatbots such as ChatGPT in its favor on issues like Gaza and the IDF, by publishing and feeding pro-Israeli articles into the large language models (LLMs) behind these chatbots. The challenge for companies developing AI models is to maintain the high level of reliability of their products, and to successfully protect them from attempts at interference and influence.
