U.S. military aircraft were airborne this spring when officials made a critical discovery. The intelligence driving an armed operation against a Chinese vessel was a hallucination generated by an AI chatbot. Officials aborted the operation at the last minute. This intervention narrowly averted a potential conflict with China, as CNN reported on Friday.
This close call highlights mounting anxiety among military leaders and external experts. As leaders rely more heavily on artificial intelligence, system errors can move up the chain of command before anyone questions them.
The incident occurred during the war with Iran. The false intelligence report claimed the targeted ship carried components destined for a nuclear weapons program.
A Special Operations Command analyst started the chain of events. The analyst queried an AI chatbot to synthesize open source data with classified signals intelligence. The chatbot mistakenly identified the cargo manifest of the ship.
The analyst then used the same AI tool a second time. This second query formatted the incorrect findings into an official summary. The document then circulated widely across military command channels.
This near miss happened while the U.S. military rushes to adopt AI. The Pentagon wants to accelerate decision-making to maintain a strategic edge over China. Defense officials view artificial intelligence as a way to speed up the kill chain so commanders respond efficiently.
However, that same operational speed introduces severe risks. Without sufficient human oversight, AI hallucinations can propagate rapidly through military networks.
Jake Steckler addressed these risks in a written statement to TechCrunch. Steckler works as a research scholar at GovAI and is a veteran U.S. Army officer. He emphasized that service members must understand the uncertainty built into large language models.
Steckler noted that understanding model uncertainty is particularly critical for decisions involving the use of force. Such decisions include targeting, intelligence analysis, and operational planning. He stressed that these choices carry life and death consequences.
Despite the severe near-miss, Steckler advocates for better safeguards rather than complete avoidance. He believes artificial intelligence tools remain useful in appropriate contexts with proper protections.
Steckler warned against prioritizing adoption speed above all else. Such an approach risks triggering incidents that destroy service member trust in the technology. Ultimately, losing that trust will only slow down long-term adoption.
