Unlocking Potential of Generative Large Language Models for Adverse Drug Reaction Relation Prediction in Discharge...

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing for Healthcare
• Pharmacovigilance

Background:

• Predicting drug-adverse event (DAE) relationships from clinical text is crucial for patient safety.
• Generative large language models (LLMs) show promise for information extraction tasks.
• Existing fine-tuned models require extensive training for DAE prediction.

Purpose of the Study:

• To comparatively analyze generative LLMs for predicting DAE relationships in discharge summaries.
• To evaluate the performance of state-of-the-art generative LLMs against fine-tuned models.
• To develop and assess a hybrid approach integrating generative LLMs with fine-tuned models for improved DAE prediction.

Main Methods:

• Comparative analysis of generative LLMs (Gemini 1.5 Pro, Llama 3.1 405B) and a fine-tuned model (BioM-ELECTRA-Large) on MIMIC-Unrestricted dataset.
• Development of a hybrid model combining BioM-ELECTRA-Large for segment selection and generative LLMs for prediction.
• Evaluation metrics included F1 score and recall for DAE relationship identification.

Main Results:

• Generative LLMs achieved high recall, comparable to fine-tuned models without specific DAE training.
• Gemini 1.5 Pro and Llama 3.1 405B outperformed BioM-ELECTRA-Large in F1 score and recall.
• The hybrid model demonstrated improved F1 scores and significant computational savings by reducing the number of segments processed by generative LLMs.

Conclusions:

• Generative LLMs are effective recall-optimized models for DAE relationship prediction.
• A hybrid approach offers a computationally efficient method for enhancing DAE prediction accuracy.
• Integrating generative LLMs with fine-tuned models holds significant potential for improving pharmacovigilance and patient safety.