Evaluating NLP Approaches to Extract Drug Indications

Area of Science:

• Natural Language Processing (NLP)
• Pharmacovigilance
• Biomedical Informatics

Background:

• Extracting reliable drug-indication knowledge is crucial for clinical decision support and pharmacovigilance.
• Manual curation of drug indications is labor-intensive and difficult to scale, necessitating automated approaches.
• FDA Structured Product Labels are a key source for drug indication information.

Purpose of the Study:

• To evaluate and benchmark nine NLP approaches for extracting therapeutic indications from FDA Structured Product Labels.
• To compare the performance of general-purpose and biomedical-specialized large language models (LLMs) against traditional methods.
• To establish a performance benchmark for LLM-based indication extraction.

Main Methods:

• Evaluated nine NLP approaches: dictionary-based matching (QuickUMLS), a biomedical-pretrained transformer (PubMedBERT), and seven LLMs.
• Benchmarked against 1,838 manually curated indication statements for twenty common medications.
• Assessed performance using metrics including F1-Score, analyzing variations by drug indication complexity.

Main Results:

• General-purpose LLMs outperformed biomedical-specialized LLMs and dictionary-based methods.
• Gemma2 (2B parameters) achieved the highest F1-Score (0.568), demonstrating strong performance despite its size.
• Dictionary-based matching (QuickUMLS) resulted in a low F1-Score (0.106) due to excessive false positives.
• Extraction accuracy varied significantly by drug, with narrow indications performing better than broad or symptom-adjacent ones.

Conclusions:

• General-purpose LLMs represent a viable and high-performing approach for automated drug-indication extraction.
• Biomedical-specific LLMs did not outperform general-purpose models in this task.
• Hybrid NLP pipelines combining LLMs with precision-oriented validation may offer optimal performance for indication extraction.