Enhancing fairness and standardization in AI-versus-physician diagnostic comparisons: A scoping review

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Clinical Research Methodology

Background:

• Direct comparisons between artificial intelligence (AI) and physicians in diagnostic tasks frequently overlook methodological rigor, focusing instead on performance outcomes.
• A significant body of literature exists comparing AI diagnostic capabilities to those of human physicians.
• Evaluating the methodological quality of these comparisons is crucial for understanding AI's true clinical utility.

Purpose of the Study:

• To critically appraise the methodological quality of studies directly comparing AI and physicians in diagnostic tasks.
• To identify key challenges and limitations in the current research landscape.
• To propose a framework for enhancing the fairness, standardization, and clinical relevance of future AI-physician comparisons.

Main Methods:

• A systematic literature search was conducted across PubMed, Scopus, and Web of Science for studies published between January 1, 2020, and October 31, 2025.
• The search adhered to PRISMA-ScR guidelines, screening 8,851 records to identify 120 studies meeting inclusion criteria for direct AI-physician comparison.
• Data extraction and narrative synthesis focused on study characteristics, dataset quality, task design, physician configuration, and reporting transparency.

Main Results:

• Significant methodological heterogeneity was observed across the 120 studies reviewed.
• Common issues include a predominance of retrospective designs (75.8%), information asymmetry (20.8%), limited clinical relevance in task design, and small physician sample sizes (60.8% with ≤10 readers).
• Widespread neglect of time constraints (50.8% of studies) and a lack of transparency in code and data availability were also identified.

Conclusions:

• Methodological weaknesses in current AI-physician diagnostic comparison studies undermine the validity and generalizability of findings.
• Future research must prioritize prospective designs, equitable experimental conditions, and enhanced transparency for reliable evidence generation.
• The proposed AI vs. Physician Study Checklist (AIPSC) aims to guide the design and reporting of more robust evaluations, supporting responsible AI integration.