LMU Munich, Department of
Abstract
Human-AI collaboration increasingly drives decision-making across industries,
from medical diagnosis to content moderation. While AI systems promise
efficiency gains by providing automated suggestions for human review, these
workflows can trigger cognitive biases that degrade performance. We know little
about the psychological factors that determine when these collaborations
succeed or fail. We conducted a randomized experiment with 2,784 participants
to examine how task design and individual characteristics shape human responses
to AI-generated suggestions. Using a controlled annotation task, we manipulated
three factors: AI suggestion quality in the first three instances, task burden
through required corrections, and performance-based financial incentives. We
collected demographics, attitudes toward AI, and behavioral data to assess four
performance metrics: accuracy, correction activity, overcorrection, and
undercorrection. Two patterns emerged that challenge conventional assumptions
about human-AI collaboration. First, requiring corrections for flagged AI
errors reduced engagement and increased the tendency to accept incorrect
suggestions, demonstrating how cognitive shortcuts influence collaborative
outcomes. Second, individual attitudes toward AI emerged as the strongest
predictor of performance, surpassing demographic factors. Participants
skeptical of AI detected errors more reliably and achieved higher accuracy,
while those favorable toward automation exhibited dangerous overreliance on
algorithmic suggestions. The findings reveal that successful human-AI
collaboration depends not only on algorithmic performance but also on who
reviews AI outputs and how review processes are structured. Effective human-AI
collaborations require consideration of human psychology: selecting diverse
evaluator samples, measuring attitudes, and designing workflows that counteract
cognitive biases.
AI Insights - Confirmation bias makes reviewers accept AI suggestions that confirm their beliefs, even if wrong.
- Intrinsic motivation dampens the effect of monetary incentives, reducing overreliance on algorithmic output.
- Simpler tasks amplify the influence of first impressions, making quality of AI suggestions more critical.
- Perceived fairness in pay can either curb or reinforce cognitive shortcuts during error correction.
- Transparent AI explanations improve error detection, offering a lever to counteract confirmation bias.
- Cognitive Illusions: A Handbook on Fallacies and Biases in Thinking, Judgement and Memory provides a taxonomy for these biases.
- Anchoring Bias Affects Mental Model Formation and User Reliance in Explainable AI Systems shows how early cues shape trust.
University of Oxford, Al
Abstract
Large language models (LLMs) distinguish themselves from previous
technologies by functioning as collaborative "thought partners," capable of
engaging more fluidly in natural language. As LLMs increasingly influence
consequential decisions across diverse domains from healthcare to personal
advice, the risk of overreliance - relying on LLMs beyond their capabilities -
grows. This position paper argues that measuring and mitigating overreliance
must become central to LLM research and deployment. First, we consolidate risks
from overreliance at both the individual and societal levels, including
high-stakes errors, governance challenges, and cognitive deskilling. Then, we
explore LLM characteristics, system design features, and user cognitive biases
that - together - raise serious and unique concerns about overreliance in
practice. We also examine historical approaches for measuring overreliance,
identifying three important gaps and proposing three promising directions to
improve measurement. Finally, we propose mitigation strategies that the AI
research community can pursue to ensure LLMs augment rather than undermine
human capabilities.
AI Insights - Explanations reduce overreliance, but they do not fully prevent decision errors.
- Uncertainty highlighting in code completions improves collaboration, yet requires careful design.
- Trust in ML models is complex; accuracy alone does not dictate user confidence.
- LLMs often fall short of human expectations, highlighting the need for capability audits.
- Papers 112, 121, 123, 128, 129, 131, 133 explore overreliance and mitigation.
- Human‑AI collaboration: a bidirectional loop where humans and models iteratively refine decisions.
- Uncertainty highlighting surfaces low‑confidence predictions to prompt human oversight.