Machine learning improves online self-referral for inflammatory rheumatic diseases: a registry-based validation study

Referral delays and errors in inflammatory rheumatic diseases (IRDs) are common. RhePort is a German online self-referral tool that estimates IRD probability based on a structured questionnaire and expert-derived algorithm. This study evaluated whether machine learning could improve IRD discrimination of RhePort. In this retrospective internal validation study, 1,333 unique RhePort questionnaires linked to rheumatologist-confirmed diagnoses were analyzed. A shared stratified five-fold cross-validation design was used for Logistic Regression, Neural Network, XGBoost, and LightGBM models. For each algorithm, feature selection and Bayesian hyperparameter optimization were performed, followed by evaluation of ensemble strategies. Discrimination, calibration, clinically relevant operating points, and SHAP-based interpretability were assessed against the original RhePort score. IRD prevalence was 35.6% (475/1,333). LightGBM achieved the strongest individual performance, with an area under the receiver operating characteristic curve of 0.791 using 30 features. A weighted ensemble of LightGBM and Logistic Regression further improved performance to 0.815 and achieved the lowest Brier score (0.166). At 95% sensitivity, specificity increased from 9% for RhePort to 32%; at 90% sensitivity, specificity increased from 15% to 48%. The most influential predictors included C-reactive protein, painful small joints, dactylitis pattern, sex, and erythrocyte sedimentation rate. Ensemble AUC-ROC was 0.828 among 746 patients with CRP and/or ESR information and 0.786 among 587 patients without these laboratory data. Machine learning improved IRD classification over the current expert-derived RhePort score and identified questionnaire items that may be omitted without loss of performance, demonstrating the potential of machine learning to enhance rheumatology referral.