Interpretable machine learning for severity classification of thyroid eye disease using orbital anatomical features

Thyroid eye disease (TED) severity assessment using the EUGOGO severity classification is partly subjective and prone to interobserver variability. While MRI-derived anatomical measurements offer objective features, such as ocular protrusion and extraocular muscle thickness, these are underutilized in machine learning (ML) models that often rely on non-interpretable radiomic features. Moreover, the inclusion of longitudinal scans from the same patient may artificially inflate model performance due to unaccounted intra-patient correlations and temporal redundancy.

To develop an interpretable machine learning framework for objective TED severity stratification (mild, moderate-to-severe, sight-threatening) by quantitatively integrating established orbital anatomical parameters aligned with clinical assessment criteria, and to evaluate how data handling strategies influence model generalizability.

We retrospectively analyzed 443 patients with TED, yielding 1,054 orbital MRI units from all available examinations. Two datasets were constructed: Dataset A included all available scans, while Dataset B retained only the first-visit orbital MRI units per patient (886 orbital units) to reduce temporal and repeat-visit bias. As bilateral orbits from the same patient were included, Dataset B reduces visit-related confounding but retains inherent inter-eye correlations. Six ML models-Logistic Regression (LR), Support Vector Machine (SVM), K-Nearest Neighbors (KNN), Random Forest (RF), XGBoost, and LightGBM-were trained and evaluated using cross-validation. Model performance was compared in terms of area under the receiver operating characteristic curve (AUC), F1-score, and recall. Feature importance was assessed using RF, XGBoost, and LightGBM.

Under class-imbalance strategies, Random Forest with class weighting achieved the highest AUC (0.811). Random Forest with SMOTE achieved the highest recall (0.669), F1-score (0.648), and specificity (0.815). Performance on Dataset A demonstrated how unaccounted longitudinal scan correlations can inflate metrics, reinforcing the necessity of temporal deduplication and cautious handling of orbital-level correlations. Feature importance analysis consistently ranked ocular protrusion as the top predictor, followed by rectus muscle thicknesses and orbital geometric parameters.

Controlling for longitudinal redundancy and intra-patient correlations significantly impacts model evaluation and generalizability. Random Forest with class weighting demonstrated the best discriminative performance in our internal validation on temporally deduplicated first-visit scans. Rather than relying on isolated diagnostic thresholds, the framework integrates measurable anatomical parameters to generate predictions that complement categorical clinical grading. This approach emphasizes standardized quantification and workflow reproducibility, highlighting the need for rigorous data structuring and transparent feature alignment in medical AI.