Predicting Penicillin Allergy: A United States Multicenter Retrospective Study

Area of Science:

• Clinical Informatics
• Allergy and Immunology
• Machine Learning in Healthcare

Background:

• Previous studies utilized non-US data to predict penicillin allergy using reaction history with logistic regression and machine learning (ML).
• The clinical utility of ML models for predicting positive penicillin allergy testing requires validation with diverse, real-world US data.

Purpose of the Study:

• To develop and evaluate ML models for predicting positive penicillin allergy skin testing using multisite US patient data.
• To identify key risk drivers influencing positive penicillin allergy test results through explainable AI methods.

Main Methods:

• A retrospective analysis of 4777 patients from 4 US hospitals was conducted, creating enriched and nonenriched training/testing datasets.
• Gradient-boosted ML models were developed and evaluated using area under the curve (AUC).
• The Shapley Additive exPlanations (SHAP) framework was employed to interpret model predictions and identify significant risk factors.

Main Results:

• The gradient-boosted model achieved an AUC of 0.67, which improved to 0.87 with complete data.
• Significant missingness was observed in key variables like reaction onset (71%), symptoms (13%), and treatment (31%).
• Top predictors for positive penicillin allergy testing included recent reactions, reactions requiring medical attention, female sex, and hives/urticaria.

Conclusions:

• ML models for predicting positive penicillin allergy testing using US retrospective data did not meet performance thresholds for clinical adoption.
• The most predictive factors identified were time since reaction, seeking medical attention, female sex, and hives/urticaria, highlighting potential areas for targeted evaluation.