Can Natural Language Processing Improve the Efficiency of Vaccine Adverse Event Report Review?

Area of Science:

• Pharmacovigilance
• Medical Informatics
• Natural Language Processing

Background:

• Spontaneous adverse event (AE) report review is crucial for medical product safety surveillance.
• The Vaccine Adverse Event Text Miner (VaeTM) was developed for automated information extraction from unstructured AE data.
• VaeTM aims to accelerate AE report evaluation and improve signal detection.

Purpose of the Study:

• To evaluate the impact of VaeTM's information extraction on the accuracy of medical expert reviews of vaccine adverse event reports.
• To compare the interpretation of VaeTM-extracted data with the interpretation of full-text reports by clinicians.

Main Methods:

• VaeTM extracted "outcome of interest," "onset time," and "alternative explanations" from 1000 VAERS reports.
• Two clinicians reviewed VaeTM output and full text, with a third clinician scoring agreement.
• Match rates and review times were analyzed.

Main Results:

• High agreement was found for "outcome of interest" (93%) and "alternative explanation" (78%).
• Time-to-onset extraction showed lower agreement (54%) unless supported by structured data (79%).
• VaeTM reduced review time by 58% (50 seconds per report) due to a 74% word reduction.

Conclusions:

• VaeTM-extracted data showed high clinical agreement with full text for key variables, despite significant text reduction.
• Further development is required for accurate extraction of time-to-onset data.
• VaeTM shows potential for improving review efficiency, but its suitability for routine use requires further investigation.