Classifying voice disorders for machine learning: a pilot study using the USVAC-C2025 diagnostic framework

Machine learning for voice disorders relies heavily on accurate diagnostic classification, yet progress has been limited by inconsistent labelling and the absence of a reproducible framework suitable for clinical and computational use. This study aimed to develop and evaluate a multilayer classification system for voice disorder diagnosis tailored for machine learning applications, and to determine its inter- and intra-rater reliability among otolaryngologists and speech-language pathologists.

We conducted a diagnostic reliability study of 45 adults with voice disorders who underwent comprehensive clinical assessment, including videostroboscopy, at a tertiary voice clinic in Sydney, Australia, between February 2018 and March 2024. A multidisciplinary team developed a five-level hierarchical classification framework through iterative consensus. Four blinded raters independently applied the framework to anonymised video and clinical datasets, with 15 cases randomly repeated for intra-rater analysis. Reliability was quantified using Fleiss κ statistics and intraclass correlation coefficients across all diagnostic levels.

Intra-rater reliability was high (intraclass correlation coefficient range, 0.768-0.865), with comparable consistency across disciplines. Inter-rater reliability was strongest for identifying disordered vs. non-disordered voices (κ = 0.812; 95% CI, 0.733-0.891) and major aetiological categories (κ = 0.695; 95% CI, 0.611-0.779), supporting the utility of structured classification for foundational diagnostic decisions. Agreement declined with increasing diagnostic specificity, particularly for perceptually based conditions such as muscle tension disorders (κ = 0.253; 95% CI, 0.172-0.334) and vocal fold paresis (κ = 0.238; 95% CI, 0.155-0.321). Functional neurological voice disorders and structural lesions demonstrated the highest category-level agreement.

These findings show that a structured, multilayer framework improves diagnostic consistency where machine learning systems most rely on stable labels and highlights key areas of diagnostic ambiguity. The system provides a practical foundation for creating reliable annotated datasets and supports future development of machine learning tools for voice disorder classification and clinical decision support.