Machine learning improves online self-referral for inflammatory rheumatic diseases: a registry-based validation study

Area of Science:

• Rheumatology
• Medical Informatics
• Machine Learning

Background:

• Referral delays and diagnostic errors are significant challenges in managing inflammatory rheumatic diseases (IRDs).
• The RhePort tool, a German online self-referral system, utilizes an expert-derived algorithm to estimate IRD probability.
• Improving the discriminative accuracy of tools like RhePort is crucial for timely patient referral and management.

Purpose of the Study:

• To evaluate the potential of machine learning (ML) algorithms to enhance the diagnostic discrimination of inflammatory rheumatic diseases (IRDs) within the RhePort self-referral tool.
• To compare the performance of various ML models against the original expert-derived RhePort score.

Main Methods:

• A retrospective analysis of 1,333 unique RhePort questionnaires with confirmed rheumatologist diagnoses was conducted.
• Logistic Regression, Neural Network, XGBoost, and LightGBM models were trained and validated using a stratified five-fold cross-validation design.
• Ensemble strategies combining ML models were evaluated, alongside feature selection and hyperparameter optimization, assessing discrimination, calibration, and interpretability (SHAP).

Main Results:

• The LightGBM model achieved an area under the receiver operating characteristic curve (AUC-ROC) of 0.791.
• A weighted ensemble of LightGBM and Logistic Regression improved performance to an AUC-ROC of 0.815, with the lowest Brier score (0.166).
• At 90% sensitivity, specificity increased from 15% with RhePort to 48% with the ML ensemble, identifying key predictors like C-reactive protein and dactylitis.

Conclusions:

• Machine learning models, particularly ensembles, significantly enhance the classification accuracy of inflammatory rheumatic diseases compared to the original RhePort score.
• ML identified redundant questionnaire items, suggesting potential for streamlining the RhePort tool without compromising performance.
• These findings highlight the potential of ML to optimize rheumatology referral pathways and improve early detection of IRDs.