Development and evaluation of a lightweight large language model chatbot for medication enquiry

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Natural Language Processing

Background:

• Large Language Models (LLMs) show potential for digital health but face computational and data security challenges.
• Limited internet accessibility in certain regions restricts the deployment of large-scale AI models.
• There is a need for efficient, domain-specific LLMs for healthcare applications.

Purpose of the Study:

• To develop and evaluate a lightweight, medical domain-specific LLM chatbot named Med-Pal.
• To address computational constraints and data security concerns in digital health applications.
• To create a robust evaluation framework for clinical LLM performance.

Main Methods:

• Fine-tuning five open-source, lightweight LLMs (≤7 billion parameters) on a curated medication-enquiry dataset (1,100 Q&A pairs).
• Validating LLMs on 231 medication-related enquiries.
• Introducing SCORE, a novel evaluation criterion for clinical adjudication by a multidisciplinary expert team.
• Selecting the best-performing LLM as Med-Pal, implementing safety guardrails against adversarial prompts.

Main Results:

• Med-Pal achieved 71.9% high-quality responses on a separate testing dataset.
• Med-Pal demonstrated superior performance compared to Biomistral and Meerkat.
• The developed SCORE framework enabled effective clinical evaluation of LLM responses.

Conclusions:

• Med-Pal's lightweight architecture and clinical alignment make it suitable for diverse healthcare settings, including those with limited digital infrastructure.
• The study highlights the feasibility of developing effective, specialized LLMs for medical applications.
• Guard-railing and expert evaluation are crucial for safe and reliable clinical LLM deployment.