Performance of Extractive and Generative Artificial Intelligence in Autonomous Systematic Reviews: A Comparative Evaluation with Published Human Reviews in Orthopaedics

There is increasing interest in artificial intelligence (AI) to accelerate selected steps of evidence synthesis. However, AI-generated syntheses should not be confused with systematic reviews (SRs) conducted according to rigorous, exhaustive, and reproducible methodology. The reliability of specialized extractive AI tools for reproducing selected SR-related tasks remains insufficiently validated. The purpose of this study was to evaluate the ability of AI tools to assist selected tasks derived from published orthopedic systematic reviews: (1) answering review-derived research questions, (2) summarizing conclusions, (3) retrieving and citing references concordant with the source SRs, organize accurately reference the scientific literature, (4) generating a PICO framework from a study title alone, and (5) extracting PICO elements from full-texts SRs.

We hypothesized that AI tools would be able to assist selected synthesis tasks, including answering review-derived questions and structuring PICO elements, but would show important limitations for bibliographic exhaustiveness, methodological reproducibility, and source concordance.

Ten peer-reviewed systematic reviews (SRs) published between 2020 and 2022 were selected as the human reference corpus. They were used as a pragmatic benchmark for comparison. Human reviewers independently extracted the main research questions and conclusions. Consensus and SciSpace were evaluated for extractive synthesis and bibliographic concordance; ChatGPT-5.1 was evaluated for PICO generation from titles and PICO extraction from full texts. Standardized prompts, scoring procedures, and he dates of AI querying were recorded.

A total of 126 AI-generated responses per tool were evaluated. Consensus achieved higher Likert scores than SciSpace (22.06 ± 2.12 vs 21.04 ± 2.76 (p = 0.001)), particularly for clarity/coherence (p = 0.003) and readability (p < 0.001), with no differences in scientific accuracy, originality, or writing efficiency. Multivariate analyses confirmed significant effects of tool, question type, cognitive level, and evaluator (all p < 0.001). Interobserver agreement was excellent (ICC (1,k) = 0.990, 95% CI: 0.989-0.992). Objective quality scores were similar between Consensus and human texts (70.52 ± 10.14 vs 69.95 ± 10.75 (p = 0.804)), whereas SciSpace scored lower (65.62 ± 8.30 (p = 0.042)) and produced shorter texts with fewer sentences (p < 0.05). SciSpace retrieved more references than Consensus (296/1,126; 26.3% vs 204/1,126; 18.1%) and showed higher concordance with gold-standard reviews (41/947; 4.3% vs 14/882; 1.6%), although overall accuracy remained low. PICO generation showed strong reliability for identifying the main objective, intervention, primary outcome, and population. PICO extraction was most reliable for population, outcomes, and main objective, but less accurate for comparator, study design, and especially inclusion period.

Specialized AI tools can assist rapid synthesis of selected questions derived from orthopedic systematic reviews, but they cannot currently replace a full systematic review or meta-analysis. Their bibliographic concordance with human SRs remained very low, confirming that narrative plausibility and true source retrieval are distinct capabilities.

IV; Methodological comparative study.