Related Experiment Videos
Evaluating the Methodological Quality of Artificial Intelligence-Assisted Systematic Reviews: Protocol for a Mixed
Mohammad Jay1,2, Mary Morgan3, Sharon Elizabeth Straus2,4,5
1Department of Medicine, Division of Endocrinology, University of Toronto, 2075 Bayview Avenue, Toronto, ON, M4N 3M5, Canada, 1 416-480-6705, 1 416-480-5761.
JMIR Research Protocols
|May 14, 2026
Summary
This study compares AI-assisted and traditional systematic reviews (SRs), evaluating AI
Area of Science:
- Meta-research and scientific methodology evaluation.
- Application of artificial intelligence (AI) and large language models (LLMs) in scientific research.
Background:
- Artificial intelligence (AI), including large language models (LLMs), is increasingly used in systematic review (SR) workflows.
- Potential benefits of AI in SRs include accelerated searching, screening, data extraction, and reporting.
- Uncertainty remains regarding AI's impact on SR methodological quality, reporting completeness, transparency, and reproducibility, compounded by inconsistent disclosure of AI use.
Purpose of the Study:
- To compare the methodological quality of AI-assisted versus traditional SRs.
- To refine, finalize, and apply an AI Transparency and Disclosure Index (AITDI).
- To evaluate the reproducibility of AI-assisted SRs and explore knowledge user perspectives on AI in SRs.
Main Methods:
- A 4-phase mixed methods meta-research study involving a matched cohort analysis of SRs published from 2023-2025.
- Evaluation of methodological quality, reporting completeness, and risk of bias using established tools (AMSTAR 2, PRISMA 2020, ROBIS).
- Application of a refined AITDI, assessment of reproducibility through comparative outputs, and analysis of qualitative interviews with knowledge users.
Main Results:
- Study preregistration and search strategy finalization completed as of December 2025.
- Title/abstract screening initiated, with data extraction planned for March-May 2026.
- AITDI refinement, reproducibility testing, and qualitative interviews are scheduled through February 2027, with final analyses by April 2027.
Conclusions:
- This study will offer one of the first empirical comparisons of AI-assisted versus traditional SRs in the LLM era.
- Findings will guide responsible AI integration and inform the development of best practices for reporting and methodology.
- Potential for developing AI-specific extensions for reporting guidelines (e.g., PRISMA-LLM, AMSTAR-LLM).
