Large language models show Dunning-Kruger-like effects in multilingual fact-checking.

Area of Science:

• Artificial Intelligence
• Computational Linguistics
• Information Science

Background:

• The proliferation of misinformation necessitates scalable and dependable fact-checking technologies.
• Large language models (LLMs) offer potential for automated fact verification, but their global efficacy is unproven.

Purpose of the Study:

• To systematically evaluate the performance of nine diverse LLMs on a multilingual dataset of 5,000 fact-checking claims.
• To assess LLM generalizability on post-training data and the impact of different prompting strategies.

Main Methods:

• Evaluation of nine LLMs across various categories (open/closed-source, size, architecture, reasoning).
• Utilized 5,000 claims fact-checked by 174 organizations in 47 languages, with over 240,000 human annotations.
• Tested model performance on claims postdating training cutoffs and four distinct prompting strategies.

Main Results:

• A Dunning-Kruger effect was observed: smaller models exhibited high confidence but low accuracy, while larger models showed higher accuracy with lower confidence.
• Significant performance disparities emerged for non-English languages and claims from the Global South, potentially exacerbating information inequalities.
• Model generalizability and prompting strategies influenced accuracy, with notable variations across LLM types and languages.

Conclusions:

• Current LLM performance in fact-checking risks systemic bias, particularly disadvantaging resource-constrained organizations and widening global information gaps.
• Established a multilingual benchmark for LLM fact-checking, emphasizing the need for equitable AI development and deployment.
• Findings provide an evidence base for policy interventions to ensure fair access to reliable AI-assisted fact-checking.