Development of a liver disease-specific large language model chat interface using retrieval-augmented generation.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Clinical Decision Support

Background:

• Commercially available Large Language Models (LLMs) are not optimized for clinical applications and may generate inaccurate information.
• Retrieval-augmented generation (RAG) is a method to customize LLMs with specific data, potentially reducing errors.
• Developing specialized LLMs is crucial for safe and effective clinical information processing.

Purpose of the Study:

• To develop a liver disease-specific LLM, named LiVersa, using RAG.
• To evaluate LiVersa's performance in accuracy, comprehensiveness, and safety compared to existing LLMs and human trainees.
• To demonstrate the feasibility of using RAG for creating protected health information-compliant clinical LLMs.

Main Methods:

• Developed LiVersa by integrating institutional protected health information-compliant text embedding and LLM platform with RAG.
• Incorporated 30 American Association for the Study of Liver Diseases guidance documents into LiVersa.
• Evaluated LiVersa by comparing its performance against trainees on a knowledge assessment and against ChatGPT 4 and Meta AI 2 using hepatologist assessments.

Main Results:

• LiVersa correctly answered all 10 questions in the knowledge assessment, outperforming trainees.
• Hepatologists rated LiVersa's outputs as more accurate than ChatGPT 4 and Meta AI 2.
• LiVersa's outputs were rated as less comprehensive and safe compared to ChatGPT 4.

Conclusions:

• Retrieval-augmented generation can be used to build disease-specific, protected health information-compliant LLMs.
• The LiVersa prototype shows promise for clinical applications, demonstrating high accuracy in hepatology.
• Further refinement is needed for LiVersa's comprehensive and safety aspects before potential clinical deployment.