Can artificial intelligence outperform experts in assessing clinical skills? Evidence from a comparative experiment

This study evaluates and compares the performance of Artificial Intelligence Assessment (AIA) and Clinical Expert Assessment (CEA) in assessing clinical skills.

Data were collected during two clinical skills competitions held in November 2024 and April 2025. Students' skills were assessed on history-taking, endotracheal intubation, and cardio-pulmonary resuscitation (CPR) by CEA and AIA at the same time. Statistical analyses included intraclass correlation coefficient (ICC), Pearson's correlation, paired-sample t-tests, and Bland-Altman plots.

For history-taking, endotracheal intubation, and CPR, scores of AIA and CEA showed moderate, excellent, and poor correlation, respectively. In history-taking, the results of ICC, Pearson's correlation, and paired-sample t-tests (CEA-AIA) are 0.40, 0.57 and 8.23. In endotracheal intubation, the results are 0.92, 0.96 and -1.31. In CPR, the results are 0.20, 0.25 and 3.75.

The findings indicate that while AIA is good at objective structured content assessments, it struggles with subjective or context-rich scenarios. Conversely, CEA provides valuable holistic judgments but is likely to have bias and lack continuous attention. Combining both methods could leverage their respective strengths, improving evaluation reliability and educational outcomes.

Integrating AIA for quantifying technical skills alongside CEA for comprehensive evaluation can enhance medical education assessments. Nonetheless, significant calibration and validation efforts are required before AI systems can be fully implemented.