Evaluation of AI Tools Versus the PRISMA Method for Literature Search, Data Extraction, and Study Composition in...

Area of Science:

• Ophthalmology
• Medical Informatics
• Artificial Intelligence in Research

Background:

• Artificial intelligence (AI) is increasingly utilized in scientific research for data analysis and manuscript preparation.
• Its application in systematic literature reviews (SLRs) is an emerging area of interest.
• Evaluating AI's efficacy in SLRs is crucial for understanding its potential and limitations.

Purpose of the Study:

• To investigate the role and limitations of AI in conducting systematic literature reviews (SLRs).
• To assess AI's performance across key SLR stages: research, study selection, data extraction, and composition.
• To compare AI-assisted SLR processes against the established Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) methodology.

Main Methods:

• Four AI platforms (Connected Papers, Elicit, ChatPDF, Jenni AI) were evaluated.
• The platforms were tested on their ability to reproduce four PRISMA-based, glaucoma-related SLRs.
• Performance was assessed for research, data extraction accuracy, and content composition.

Main Results:

• AI tools did not fully replicate PRISMA's comprehensive results in literature searching.
• Data extraction accuracy varied: Elicit averaged 51.40% accuracy, ChatPDF 60.33%.
• Jenni AI demonstrated fluency but lacked methodological and result-elaboration depth.

Conclusions:

• The PRISMA method remains the gold standard for SLR accuracy and reproducibility.
• AI tools can assist with repetitive tasks but cannot replace researcher critical oversight.
• Maintaining researcher involvement is essential for ensuring the quality, accuracy, and objectivity of SLRs.