Assessing the methodologic quality of systematic reviews using generative large language models.

Area of Science:

• Artificial Intelligence in Medicine
• Medical Informatics
• Urology Research

Background:

• Assessing the methodological quality of systematic reviews (SRs) is crucial for evidence-based medicine.
• Generative large language models (LLMs) offer potential for automating complex analytical tasks.

Purpose of the Study:

• To evaluate the accuracy of generative LLMs in assessing the methodological quality of urological SRs.
• To compare LLM-based quality assessment with human expert penilaian.

Main Methods:

• 114 urological SRs were assessed by human experts and a customized GPT model.
• GPT underwent three zero-shot assessment iterations and an enhanced trial using chain-of-thought prompting.
• Performance metrics included accuracy, sensitivity, specificity, and F1 score against human judgments.

Main Results:

• GPT achieved 75% overall congruence with human reviewers, with 77% for critical criteria.
• The average F1 score was 0.66, and internal validity was high at 85%.
• Enhanced prompting improved critical criteria congruence to 91% and overall accuracy to 93%.

Conclusions:

• Generative LLMs demonstrate a promising capacity for efficient and accurate quality assessment of SRs in urology.
• LLM-based tools can potentially streamline the review process and support evidence synthesis.