A study by researchers in France and Italy reveals that providing people with AI-generated answers drastically lowers their accuracy and willingness to admit uncertainty, while paradoxically boosting their confidence.
- Users much less likely to admit ignorance with AI advice
- Accuracy drops to one third despite rise in confidence
- AI answers often wrong, yet trusted by participants
What happened
Researchers from three universities in France and Italy conducted a study to examine how AI advice affects human decision-making and confidence. Participants were given questions designed specifically to challenge AI reliability by focusing on obscure visual details from movies. The AI model used, Step 3.5 Flash, commonly provided incorrect answers on these items.
The study found that when participants had access to AI advice, their willingness to say “I don’t know” plummeted from 44% to 3%, while their overall accuracy dropped sharply from 27% to 9%. Meanwhile, confidence levels nearly doubled, moving from 30% to 76%. Even monetary incentives failed to significantly counteract this effect.
Why it matters
The findings highlight a worrying dynamic where AI advice suppresses critical thinking and the healthy cognitive habit of acknowledging one's knowledge limits. According to the researchers, this “cognitive surrender” leads people to accept wrong AI-generated answers with high confidence, increasing the risk of misinformation and errors.
This is especially concerning for younger generations growing up with AI assistants, potentially impacting the development of foundational critical thinking skills. Technology designs that prioritize providing confident answers over admitting uncertainty can inadvertently encourage overreliance on flawed AI judgments.
What to watch next
Future research will likely explore strategies to integrate AI that better preserves human judgment, such as AI systems that admit uncertainty or encourage questioning. Educators and product designers will need to address how AI impacts learning and decision-making by fostering awareness of AI limitations.
Regulators and technology companies might also face pressure to reconsider user interfaces that eliminate options to express uncertainty, especially in education and information retrieval contexts. Monitoring how AI influences long-term trust and cognitive skills remains a critical area as AI becomes more ubiquitous in daily life.