Large language models for the screening step in systematic reviews in dentistry

This study assessed the performance of chatbots in the screening step of a systematic review (SR) with an exemplary focus on tooth segmentation on dental radiographs using artificial intelligence (AI).

A comprehensive systematic search was performed in December 2024 across seven databases: PubMed, Scopus, Web of Science, Embase, IEEE, Google Scholar, and arXiv. Five chatbots-ChatGPT-4, Claude 2 100k, Claude Instant 100k, Meta's LLaMA 3, and Gemini-were evaluated for their ability to screen articles on tooth segmentation on radiographs using AI. The evaluations took place from January to February 2025, focusing on performance metrics such as accuracy, precision, sensitivity, specificity, and F1-score for screening quality measured against expert reviewers' screening, as well as Cohen's Kappa for inter-rater agreement between different chatbots.

A total of 891 studies were screened. Significant variability in the number of included or excluded studies was observed (p < 0.001/Chi-square), with Claude-instant-100k having the highest inclusion rate (54.88 %) and ChatGPT-4 the lowest (29.52 %). Gemini excluded the most studies (67.90 %), while ChatGPT-4 marked the highest number of studies for full-text review (5.39 %). Fleiss' Kappa (-0.147, p < 0.001) indicated systematic disagreement between chatbots worse than random chance. Performance metrics varied; ChatGPT-4 had the highest precision (24 %) and accuracy (75 %) measured against human expert reviewers, while Claude-instant-100k had the highest sensitivity (96 %) but the lowest precision (16 %).

Chatbots showed limited accuracy during study screening and low inter-rater agreement. There remains the need for human oversight during systematic reviewing.

Theoretically, Chatbots can streamline SR tasks such as screening. However, human oversight remains critical to maintain the integrity of the review.