Automated Multi-Label Adverse drug Reaction Extraction from Malaysia's National Pharmacovigilance Narratives

Extracting multiple adverse drug reaction (ADR) terms from unstructured narratives remains challenging, particularly under severe label imbalance that limits the detection of rare ADRs. This study aimed to develop and evaluate a multi-label natural language processing framework for automated ADR extraction within Malaysia's national pharmacovigilance reporting system. We evaluated classical machine learning and transformer-based models within a common framework, incorporating domain-specific preprocessing and imbalance-aware optimization strategies to improve the detection of rare ADRs. The framework was applied to 28,980 real-world ADR narratives annotated with 80 Medical Dictionary for Regulatory Activities (MedDRA) Preferred Terms (PTs) from the Skin and Subcutaneous Tissue Disorders System Organ Class (SOC). Performance was evaluated using micro-F1, macro-F1, precision, recall, and label coverage using a predefined 80/20 train-test evaluation strategy, complemented by pharmacist pilot review. Transformer-based models achieved the strongest performance, with the augmented model attaining a micro-F1 of 0.88, macro-F1 of 0.55, recall of 0.88, and the highest label coverage, correctly predicting 60 of 80 PTs (75%). However, a 70/10/20 sensitivity analysis on the improved transformer model yielded lower micro-F1 and precision, highlighting the impact of further data partitioning on model performance. Improved classical models also showed substantial gains over their baseline models, particularly in micro-F1, macro-F1, recall, and label coverage, consistent with improved detection of rare ADRs. During the pilot review, pharmacists accepted the model-preselected PTs without modification in 19 of 25 narratives (76%), while additional PTs were added in the remaining cases. Although the study was limited to a single SOC and one national pharmacovigilance database, the findings demonstrate that optimization strategies, domain-specific feature enrichment, and targeted augmentation can substantially improve multi-label ADR classification under severe imbalance. These findings support further evaluation of AI-assisted pharmacovigilance with pharmacist oversight in real-world practice.