Sorting Through the Safety Data Haystack: Using Machine Learning to Identify Individual Case Safety Reports in...

Area of Science:

• Pharmacovigilance and Medical Informatics
• Computational Linguistics and Natural Language Processing

Background:

• Social digital media (SDM) offers vast data for pharmacovigilance but presents challenges due to low information content and high noise.
• Traditional pharmacovigilance workflows struggle with the volume of SDM data, straining limited human resources for adverse event identification.
• Advances in medical informatics enable automated methods for detecting valid individual case safety reports (ICSRs) within SDM.

Purpose of the Study:

• To develop and compare rule-based and machine learning (ML) models for classifying ICSRs from SDM.
• To evaluate the performance of these models against human pharmacovigilance experts.

Main Methods:

• A dataset of 311,189 social media posts mentioning Roche products was used.
• Rule-based and ML models were developed and iterated, including components for annotating ICSR elements and final validity decisions.
• Performance was measured by agreement with human experts using the Gwet AC1 statistic (gKappa).

Main Results:

• The initial rule-based model achieved 65% accuracy and 46% gKappa.
• Incorporating ML components improved performance, with the final model reaching 83% accuracy and 78% gKappa on a blind test set.
• The automated system processed 311,189 posts in 48 hours, a task estimated to take human experts 44,000 hours.

Conclusions:

• Automated ML classifiers are effective and scalable for identifying potential ICSRs in SDM.
• A hybrid workflow combining ML detection with human subject matter expert (SME) review offers an efficient solution for pharmacovigilance.