Enhancing fairness and standardization in AI-versus-physician diagnostic comparisons: A scoping review

The growing number of studies directly comparing artificial intelligence (AI) to physicians in diagnostic tasks often focuses on performance outcomes, overlooking fundamental methodological rigor. This scoping review aims to critically appraise the methodological quality of this body of literature, identifying key challenges and proposing a framework to enhance the fairness, standardization, and clinical relevance of future comparisons.

We conducted a systematic search of PubMed, Scopus, and Web of Science for studies published between January 1, 2020, and October 31, 2025, following the PRISMA-ScR guidelines. From 8,851 screened records, 120 studies met the inclusion criteria for direct AI-physician comparison. Data on study characteristics, dataset quality, task design, physician configuration, and reporting transparency were extracted and synthesized narratively.

Our analysis of 120 studies revealed a field characterized by significant methodological heterogeneity. Key issues include a predominant focus on retrospective studies (75.8%), frequent information asymmetry between AI and physicians (20.8%), limited clinical relevance in task design despite superficial fidelity, and insufficient physician sample sizes (60.8% had ≤ 10 readers). Furthermore, we found a widespread neglect of time constraints (absent in 50.8% of studies) and a critical lack of transparency regarding code and data availability.

Current research on AI-physician diagnostic comparisons is often hampered by methodological weaknesses that undermine the validity and generalizability of its findings. To ensure the generation of reliable and clinically meaningful evidence, future studies must prioritize prospective designs, ensure fairness in experimental conditions, and adhere to higher standards of transparency. We propose the AI vs. Physician Study Checklist (AIPSC) as a practical tool to guide the design and reporting of more robust and systematic evaluations, ultimately fostering the responsible integration of AI into clinical practice.