Real-world evaluation of medication recommendation workflows: Retrieval augmentation, physician-RAG collaborative...

Area of Science:

• Clinical Informatics
• Artificial Intelligence in Healthcare
• Pharmacology

Background:

• Large language models (LLMs) show potential for clinical decision support, but their effectiveness in medication recommendation is not well-established.
• Evaluating LLMs for prescribing requires assessing not just reasoning but also real-world readiness, including safety and accuracy.
• Existing evaluations often focus on benchmarks rather than practical prescribing capabilities.

Purpose of the Study:

• To compare four distinct medication recommendation workflows.
• To evaluate workflows based on a clinical framework assessing completeness, safety, ranking, and prescription accuracy.
• To determine the optimal approach for LLM-assisted medication recommendations in real-world clinical settings.

Main Methods:

• Retrospective analysis of 800 clinical cases from Shanghai East Hospital.
• Development of an expert reference standard categorizing medications as CORE, ALT (acceptable alternatives), or AVOID (unsafe).
• Comparison of four workflows: Physician, Base LLM, Retrieval-augmented LLM (RAG), and Physician-RAG collaborative, using metrics like recall, precision, and AVOID-hit rate.

Main Results:

• The Physician-RAG collaborative workflow achieved the highest recall for CORE (1.000) and ALT (0.955) therapies, and overall recall (0.978).
• This workflow also demonstrated the highest precision (0.963) and Jaccard index (0.943), with the lowest case-level AVOID-hit rate (0.026).
• Physician-RAG collaborative exhibited the highest prescription-parameter accuracy (0.951), significantly outperforming the Base LLM workflow (0.494).

Conclusions:

• Stand-alone LLM output showed lower agreement and prescribing accuracy compared to supervised workflows.
• Retrieval augmentation enhanced medication coverage and accuracy, with Physician-RAG collaborative excelling across all evaluated criteria.
• Physician-supervised collaboration is a promising direction for medication decision support, warranting prospective studies before clinical deployment.