Bio-K-Transformer: A pre-trained transformer-based sequence-to-sequence model for adverse drug reactions prediction

Adverse drug reactions (ADRs) pose a serious threat to patient health, potentially resulting in severe consequences, including mortality. Accurate prediction of ADRs before drug market release is crucial for early prevention. Traditional ADR detection, relying on clinical trials and voluntary reporting, has inherent limitations. Clinical trials face challenges in capturing rare and long-term reactions due to scale and time constraints, while voluntary reporting tends to neglect mild and common reactions. Consequently, drugs on the market may carry unknown risks, leading to an increasing demand for more accurate predictions of ADRs before their commercial release. This study aims to develop a more accurate prediction model for ADRs prior to drug market release.

We frame the ADR prediction task as a sequence-to-sequence problem and propose the Bio-K-Transformer, which integrates the transformer model with pre-trained models (i.e., Bio_ClinicalBERT and K-bert), to forecast potential ADRs. We enhance the attention mechanism of the Transformer encoder structure and adjust embedding layers to model diverse relationships between drug adverse reactions. Additionally, we employ a masking technique to handle target data. Experimental findings demonstrate a notable improvement in predicting potential adverse reactions, achieving a predictive accuracy of 90.08%. It significantly exceeds current state-of-the-art baseline models and even the fine-tuned Llama-3.1-8B and Llama3-Aloe-8B-Alpha model, while being cost-effective. The results highlight the model's efficacy in identifying potential adverse reactions with high precision, sensitivity, and specificity.

The Bio-K-Transformer significantly enhances the prediction of ADRs, offering a cost-effective method with strong potential for improving pre-market safety evaluations of pharmaceuticals.