Benchmarking Large Language Models for Drug Combination Alerts: Achieving Expert-Level Reliability via Knowledge...

Area of Science:

• Artificial Intelligence in Medicine
• Computational Pharmacology
• Clinical Decision Support Systems

Background:

• Large language models (LLMs) are increasingly explored for healthcare applications.
• The reliability of LLMs for identifying critical drug-drug interactions (DDIs) is unvalidated.
• Accurate DDI identification is crucial for patient safety and effective pharmacotherapy.

Purpose of the Study:

• To systematically evaluate LLM potential for drug combination alerting using the CoMed framework.
• To assess the impact of knowledge grounding (RAG) and expert reasoning on LLM performance.
• To demonstrate the utility of a multiagent architecture for interpretable DDI risk analysis.

Main Methods:

• Evaluated native LLM performance as a baseline.
• Integrated Retrieval-Augmented Generation (RAG) for external knowledge grounding.
• Applied context engineering for expert-guided reasoning.
• Utilized a multiagent architecture for comprehensive risk assessment.

Main Results:

• Qwen2.5-Max-CoT, integrating RAG and context engineering, achieved high performance (F1 = 0.971, AUC = 0.982).
• The model demonstrated an expert-level balance between precision and recall in DDI detection.
• A case study on aspirin-warfarin confirmed the CoMed framework's ability to generate accurate, traceable HTML reports.

Conclusions:

• Enhanced LLMs, particularly with RAG and expert reasoning, can reliably detect risky drug combinations.
• The CoMed framework facilitates transparent and traceable drug interaction risk assessment.
• These advanced LLMs show potential to support clinical decision-making in pharmacotherapy.