Drug-drug interaction identification using large language models

Area of Science:

• Pharmacology and Toxicology
• Artificial Intelligence in Medicine
• Clinical Informatics

Background:

• Drug-drug interactions (DDIs) are a major cause of patient harm, especially with multiple medications.
• Current electronic health record (EHR) systems use rules-based software to detect DDIs.
• Large language models (LLMs) offer potential for DDI identification but require rigorous validation.

Purpose of the Study:

• To benchmark the performance of LLMs in identifying and managing DDIs.
• To develop and utilize a clinician-annotated dataset for evaluating LLM DDI detection capabilities.
• To assess LLM performance across various task complexities and interaction severities.

Main Methods:

• Evaluated three LLMs (GPT-4o-mini, MedGemma-27B, LLaMA3-70B) on a 750-scenario DDI dataset.
• Utilized three task formats: two-drug classification, three-drug discrimination, and 4-6 drug selection.
• Assessed performance using precision, recall, F1 score, accuracy, self-consistency, and confidence-aligned metrics.

Main Results:

• LLaMA3-70B excelled in two-drug classification recall and F1 score.
• GPT-4o-mini demonstrated superior accuracy and consistency in multi-drug tasks.
• Model self-consistency and reliability decreased with increasing task complexity.

Conclusions:

• LLMs demonstrate variable capabilities for DDI identification, with performance degradation on complex tasks.
• Current LLMs lack uniform reliability across different reasoning formats.
• Multi-format evaluation and reliability-aware assessments are crucial for safe LLM application in medication safety.