Experts Warn of Growing Trust in AI Despite Repeated Errors
Concerns are mounting over the growing tendency of people to trust artificial intelligence systems even after repeated examples of serious mistakes, with experts warning that the problem is spreading across law, journalism, research, software development and other professions.
In April, the Alabama Supreme Court sanctioned a lawyer for filing court briefs filled with inaccurate citations generated by AI, including references to cases that never existed. According to a concurring opinion by one of the justices, the attorney continued citing fake cases even after being warned about the problem.
Another lawyer was reportedly sanctioned during the same week for repeatedly submitting AI-generated material containing false information despite earlier warnings from the court.
The incidents are part of a wider trend documented by researchers tracking AI-related errors in legal systems. Damien Charlotin, a senior research fellow at HEC Paris, maintains a database of court cases involving AI mistakes. According to him, more than 1,400 cases have been recorded globally in the past three years.
Charlotin noted that while the number of cases had previously grown rapidly, the trend has now stabilised at roughly 350 to 400 court decisions every quarter involving AI-related issues.
Because court proceedings are public, AI errors in legal filings are easier to identify. However, experts say similar mistakes are occurring in less visible industries such as media, academia and technology.
On May 19, The New York Times reported that the author of a book discussing the future of AI and truth admitted that several quotes in the publication were either fabricated or wrongly attributed by artificial intelligence tools.
Researchers say the growing issue is not simply that AI makes mistakes, but that humans continue to trust the technology even when they know it can be unreliable.
Alan Wagner, an associate professor of aerospace engineering at Pennsylvania State University, said people naturally assume machines are more knowledgeable and dependable than humans.
“Humans have a tendency to believe machines are infallible,” Wagner explained.
Studies also suggest that AI-generated information may appear more convincing than human advice, even when both are equally unreliable.
A recent study published in February found that participants who believed guidance came from AI were more likely to trust it, even when the information was wrong. Researchers observed that participants who already had positive views about AI performed worse because they relied too heavily on the system’s recommendations.
The issue could have even more serious consequences in military or emergency situations.
In one experiment inspired by drone warfare scenarios, researchers asked participants to identify whether people in images were civilians or enemy combatants. When a robot contradicted their original judgment, most participants changed their answers, even though the robot’s feedback was random.
Researchers warned that if such situations happened in real life, many innocent civilians could have been harmed because of blind trust in automated systems.
Experts say modern AI systems are especially dangerous because they can now perform a broad range of complex tasks, from writing legal documents to generating computer code, making it easier for users to surrender critical thinking to machines.
Researchers at the University of Pennsylvania’s Wharton School described the phenomenon as “cognitive surrender,” where users stop independently verifying information once AI becomes involved.
Attempts to reduce overreliance on AI through warnings and education have produced only limited success.
In one study conducted at Boston University, students were warned that ChatGPT often produces inaccurate summaries and struggles with advanced mathematics. While some participants became more careful in checking source summaries, many still failed to verify mathematical answers.
Researchers found that time pressure, confidence in AI and workplace expectations often outweighed caution.
Experts say the situation is becoming more complicated as AI systems improve in many areas. Because the technology often performs correctly, users may become less likely to question the mistakes that still occur.
“They don’t ever get to the ground truth,” researcher Sophie Nightingale said. “They continue believing the AI is correct because they never see the errors.”
