Artificial intelligence for solving pediatric clinical cases: A Retrieval-Augmented approach utilizing Llama3.2 and...

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Pediatric Informatics

Background:

• Large Language Models (LLMs) exhibit "hallucinations," impacting their reliability in pediatric care.
• The accuracy of LLMs in complex pediatric clinical scenarios is a significant concern.
• Enhancing LLM performance for pediatric applications requires robust information integration.

Purpose of the Study:

• To evaluate the efficacy of a Retrieval-Augmented Generation (RAG) system in improving LLM performance for pediatric clinical cases.
• To assess whether integrating the Nelson Textbook of Pediatrics enhances Llama3.2's accuracy.
• To determine the impact of RAG on addressing complex pediatric clinical scenarios.

Main Methods:

• Assessed RAG system performance using 1,713 multiple-choice pediatric clinical questions from MedQA and ADC-EP datasets.
• Compared Llama3.2 performance in standalone mode versus RAG-integrated mode.
• Utilized chi-square test to compare accuracy percentages, with p < 0.05 considered significant.

Main Results:

• The RAG-integrated system achieved 67.78% accuracy, significantly outperforming standalone Llama3.2 (46.18%).
• Statistical analysis showed a significant improvement with RAG integration (p = 1.5e-112).
• Accuracy enhancement was consistent across all pediatric subspecialties.

Conclusions:

• Retrieval-Augmented Generation (RAG) systems demonstrably enhance LLM accuracy in pediatric clinical contexts.
• Integrating specialized medical knowledge, like the Nelson Textbook of Pediatrics, via RAG improves LLM reliability.
• RAG systems offer a promising approach to enhance safety and reliability in AI-driven clinical decision-making for pediatrics.