Evaluation of four machine learning models for signal detection.

Area of Science:

• Pharmacovigilance and drug safety research.
• Application of machine learning in healthcare data analysis.

Background:

• Traditional disproportionality analysis for pharmacovigilance has limitations in handling confounders.
• Logistic regression offers advantages over disproportionality analysis by accounting for confounding factors.
• Machine learning algorithms present opportunities to further enhance drug safety signal detection.

Purpose of the Study:

• To compare the signal detection performance of logistic regression, gradient-boosted trees, random forest, and support vector machine models.
• To evaluate these models using data from the U.S. Food and Drug Administration Adverse Event Reporting System (FAERS).

Main Methods:

• A cross-sectional study design utilizing FAERS data from Q4 2017 to Q4 2020.
• Implementation of two training sets: one stratified on the outcome variable and another using Synthetic Minority Oversampling Technique (SMOTE) to address data imbalance.
• Development of crude and hyperparameter-tuned models for each algorithm.
• Performance evaluation using metrics such as accuracy, precision, F1 score, recall, ROCAUC, and PRCAUC.

Main Results:

• Models trained on a balanced dataset demonstrated superior accuracy, F1 score, and recall compared to those trained on SMOTE-augmented data.
• Logistic regression, gradient-boosted trees, random forest, and support vector machine models exhibited similar performance metrics when trained on the balanced dataset.
• The hyperparameter-tuned gradient-boosted trees model achieved the highest ROCAUC (0.646), while the crude random forest model yielded the highest PRCAUC (0.839) on the balanced dataset.

Conclusions:

• All evaluated machine learning models performed comparably on the balanced training set.
• Logistic regression models demonstrated higher accuracy, precision, and recall.
• Integrating disproportionality analysis results with additional case report information significantly improved model performance metrics compared to disproportionality analysis alone.