Evaluating the Utility of Artificial Intelligence in Conducting Systematic Reviews

Area of Science:

• Clinical Research
• Artificial Intelligence
• Medical Informatics

Background:

• Systematic reviews are crucial for high-level clinical evidence but are time-consuming.
• Large language models (LLMs) like ChatGPT-5 offer potential efficiency gains.
• AI's performance in full systematic review workflows needs evaluation.

Purpose of the Study:

• To compare ChatGPT-5 (Deep Research and Agent Modes) with human researchers in systematic review article retrieval.
• To assess AI's efficiency and accuracy against a gold standard in total joint arthroplasty reviews.

Main Methods:

• Five systematic reviews on total joint arthroplasty were used as a gold standard.
• Orthopaedic research fellows and ChatGPT-5 independently screened articles.
• AI searches were repeated for reproducibility; performance metrics included recall, precision, and time.

Main Results:

• Human reviewers screened 9101 articles in 268 hours, achieving 85.2% recall.
• ChatGPT-5 modes averaged 12-14 minutes per search, identifying 40.9-47.5% of articles.
• AI had more false negatives but identified additional eligible studies.

Conclusions:

• Human expertise remains essential for nuanced systematic review article selection.
• ChatGPT-5 drastically reduces search time and complements human efforts.
• AI shows promise as an adjunct tool in systematic review processes under expert supervision.