Evaluating tonsillectomy-related YouTube videos via a human expert review and the ChatGPT-4: a multi-method quality...

The quality and reliability of health-related content on YouTube remain a growing concern. This study aimed to evaluate tonsillectomy-related YouTube videos using a multi-method framework that combines human expert review, large language model (ChatGPT-4) analysis, and transcript readability assessment.

A total of 76 English-language YouTube videos were assessed. Two otolaryngologists independently rated video quality using the DISCERN instrument and JAMA benchmarks. Corrected transcripts were evaluated by ChatGPT-4 (May 2024 version) for accuracy and completeness. Spearman correlations and regression analyses were used to explore associations between human and AI evaluations. Videos were also categorized as transcript-heavy or visually rich to examine the effect of visual presentation.

Professional videos consistently outperformed patient-generated content in quality metrics. ChatGPT-4 accuracy scores showed a strong correlation with JAMA ratings (ρ = 0.56), while completeness was strongly associated with DISCERN scores (ρ = 0.72). Visually rich videos demonstrated significantly higher AI accuracy than transcript-heavy videos (Cohen's d = 0.600, p = 0.030), suggesting that visual context may enhance transcript-based interpretation. However, the average transcript readability (FKGL = 8.38) exceeded the recommended level for patient education.

Tonsillectomy-related YouTube content varies widely in quality. Human-AI alignment supports the use of large language models for preliminary content screening. Visually enriched content may improve AI interpretability, while readability concerns highlight the need for more accessible educational resources. Multimodal evaluation and design should be prioritized in future digital health content.