Not Ready for Prime Time: Limitations of a Retrieval-Augmented Generation Large Language Model in Assessing Risk of...

Area of Science:

• Artificial Intelligence in Medical Research
• Systematic Review Augmentation
• Pediatric Orthopaedics Literature Analysis

Background:

• Large language models (LLMs) are increasingly used to enhance systematic reviews.
• LLMs can "hallucinate"; retrieval-augmented generation (RAG) limits this by using provided sources.
• This study assessed a RAG-LLM's accuracy in quality appraisal of pediatric orthopaedic observational studies.

Purpose of the Study:

• To evaluate the accuracy and reliability of a RAG-LLM (NotebookLM) for quality assessment of observational studies in pediatric orthopaedics.
• To compare the RAG-LLM's performance against manual review (ground truth).

Main Methods:

• Two existing systematic reviews of pediatric orthopaedic observational studies were included.
• NotebookLM assessed study quality using the Newcastle-Ottawa Scale (NOS).
• Intraclass correlation coefficients (ICC) measured agreement between RAG-LLM scores and manual review scores.

Main Results:

• Overall agreement between RAG-LLM instances and manual review was moderate (ICC(2,k) = 0.69).
• Agreement between individual RAG-LLM scores and manual review was poor (ICC(2,1) ranging from 0.081 to 0.27).
• Percent agreement ranged from 14.8% to 29.6%.

Conclusions:

• The RAG-LLM (NotebookLM) demonstrated low reliability and accuracy in quality assessment.
• Researchers should exercise caution when using AI tools like RAG-LLMs in research.
• Independent critical appraisal by researchers remains essential for new studies.