ADE Eval: An Evaluation of Text Processing Systems for Adverse Event Extraction from Drug Labels for...

Area of Science:

• Pharmacovigilance
• Medical Informatics
• Natural Language Processing

Background:

• The U.S. Food and Drug Administration (FDA) seeks automated tools for identifying adverse drug events (ADEs) in prescribing information.
• The MITRE Corporation and FDA collaborated on the Adverse Drug Event Evaluation (ADE Eval) shared task to assess NLP algorithm performance for pharmacovigilance.

Purpose of the Study:

• To evaluate the effectiveness of various natural language processing (NLP) techniques in identifying ADEs within FDA-approved drug labels.
• To model real-world pharmacovigilance use cases and practices within the FDA.

Main Methods:

• Development of pharmacovigilance-specific annotation guidelines and corpora.
• Evaluation of algorithms using metrics reflecting FDA safety evaluator experiences: MedDRA term coding accuracy, evidence quality, and mention-finding F1-measure.

Main Results:

• Thirteen teams submitted 23 runs, achieving top scores of 0.79 for MedDRA coding F1-measure, 0.96 for quality score, and 0.89 for mention-finding F1-measure.
• The top-performing algorithms demonstrated significant progress in NLP for ADE identification.

Conclusions:

• Current NLP techniques require human oversight but show potential for integration into pharmacovigilance workflows.
• Further exploration of NLP outputs within human-assisted pharmacovigilance processes is warranted.