Chatbots for conducting systematic reviews in pediatric dentistry

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Dental Research

Background:

• Systematic reviews (SRs) are crucial for evidence-based practice.
• AI-powered chatbots offer potential to streamline complex research tasks.
• Evaluating chatbot performance in SRs is essential for their adoption.

Purpose of the Study:

• To assess the performance of AI chatbots (ChatGPT4, Gemini) in key steps of a systematic review.
• To compare chatbot performance against non-expert human reviewers and expert standards.
• To evaluate efficiency gains and limitations of chatbots in systematic review processes.

Main Methods:

• A systematic review on AI in pediatric dentistry was conducted.
• Five tasks were evaluated: PICO question formulation, search query development, study screening, data extraction, and risk of bias assessment.
• Chatbots and non-experts were compared to expert reviewers using various performance metrics.

Main Results:

• Chatbots showed high sensitivity (>90%) but lower precision (<25%) in screening.
• Performance in data extraction and PICO formulation varied, with chatbots generally outperforming non-experts.
• Agreement with expert assessments for risk of bias was comparable across chatbots and non-experts (around 50%).

Conclusions:

• AI chatbots can enhance efficiency in systematic review tasks like screening and data extraction.
• Human oversight remains critical to ensure the accuracy and reliability of AI-assisted systematic reviews.
• The effectiveness of chatbots in systematic reviews is task-dependent, requiring informed selection by researchers.