Artificial intelligence vs. human evaluation of anesthesia education videos: a comparative analysis using validated...

YouTube has become an increasingly popular platform for medical education, yet the accuracy and educational quality of anesthesia-related videos remain uncertain. While human experts have traditionally assessed video quality using validated scales such as DISCERN, JAMA, and the Global Quality Scale (GQS), artificial intelligence (AI) models-particularly large language models (LLMs)-now offer new possibilities for scalable, objective content evaluation. This study aimed to compare the educational quality of anesthesia education videos produced by humans and AI, and to examine the level of agreement between human expert ratings and ChatGPT-5 evaluations.

In this cross-sectional analytical study, forty YouTube videos were analyzed: 20 produced by human educators and 20 generated using AI tools. Each video was independently assessed by two anesthesiologists and by ChatGPT-5 Plus (OpenAI, 2025) using DISCERN, JAMA, and GQS criteria. Inter-rater reliability between human evaluators was determined using the Intraclass Correlation Coefficient (ICC), and correlations between human and AI ratings were analyzed with Spearman's rho.

Human-generated videos scored significantly higher than AI-generated ones in DISCERN (68.45 ± 4.60 vs. 62.77 ± 7.32, p = 0.0044, Cohen's d = 0.82) and JAMA (3.70 ± 0.41 vs. 3.23 ± 0.77, p = 0.0446, Cohen's d = 0.71) scores, whereas no significant difference was observed in GQS scores (p = 0.3033). Inter-rater reliability between human experts was excellent (ICC = 0.81-0.86, p < 0.001). Strong correlations were found between ChatGPT-5 and the human mean scores for all scales (ρ = 0.897 for DISCERN, ρ = 0.785 for GQS, ρ = 0.765 for JAMA; p < 0.001), indicating high agreement between AI and human evaluations.

AI-based models such as ChatGPT-5 show potential to approximate human expert judgment in evaluating educational content. While human-generated videos remain superior in terms of source transparency and ethical reporting, AI-generated content approaches human quality in structural organization and linguistic fluency. These findings suggest that AI-assisted evaluation systems may serve as standardized, efficient tools for quality screening of large-scale educational video archives in medical education.