Large Language Models as Clinical Support Tools in Drug Information Services: Performance Comparison With Pharmacists

Area of Science:

• Pharmacology
• Artificial Intelligence
• Health Informatics

Background:

• Large language models (LLMs) are increasingly utilized for drug information support.
• Concerns exist regarding the reliability and clinical applicability of LLM-generated drug information.
• This study addresses the need to evaluate LLM performance in real-world clinical settings.

Purpose of the Study:

• To evaluate the performance of multiple large language models (LLMs) in answering drug information questions.
• To assess LLM responses for clarity in Thai, concordance with pharmacist answers, relevance, context awareness, and citation credibility.
• To compare the capabilities of different LLMs in a practical healthcare context.

Main Methods:

• 102 drug information questions from a university hospital were posed to seven LLMs.
• Responses were evaluated by assessors using a predefined rubric, with pharmacist answers as the reference standard.
• Inter-rater reliability was established, and statistical tests (Cochran's Q, McNemar) were used for performance comparison.

Main Results:

• LLMs demonstrated substantial inter-rater agreement (0.79-0.86).
• High scores were observed for clarity (85.25%-92.75%), relevance (0.97-1.00), and context awareness (0.90-0.99).
• Concordance with pharmacist answers varied (0.70-0.86), and citation credibility was consistently low (0.03-0.36).

Conclusions:

• LLMs show potential for preliminary drug information retrieval and rapid response generation.
• Variable concordance and citation limitations highlight the need for continued pharmacist oversight.
• LLMs can serve as supplementary tools in drug information services, but human validation remains critical.