Optimization of hepatological clinical guidelines interpretation by large language models: a retrieval augmented...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Decision Support

Background:

• Large language models (LLMs) offer potential to revolutionize healthcare delivery.
• Integrating LLMs into clinical workflows requires optimizing information retrieval and interpretation.
• Clinical Decision Support Systems (CDSSs) can benefit from enhanced medical guideline interpretation.

Purpose of the Study:

• To investigate the integration of LLMs into healthcare, specifically for improving CDSSs.
• To enhance the accurate interpretation of medical guidelines for chronic Hepatitis C Virus (HCV) infection management.
• To evaluate the impact of framework customization, including retrieval augmented generation (RAG) and prompt engineering, on LLM performance.

Main Methods:

• Developed a customized LLM framework using OpenAI's GPT-4 Turbo with RAG and prompt engineering.
• Converted medical guidelines into a structured format for efficient LLM processing.
• Conducted an ablation study comparing baseline GPT-4 Turbo with five experimental setups, including guideline reformatting and few-shot learning.

Main Results:

• LLM accuracy in interpreting medical guidelines improved significantly from 43% to 99% (p < 0.001).
• Providing guidelines as a coherent text corpus and converting non-text sources to text were crucial for accuracy.
• Few-shot learning did not yield significant improvements in overall accuracy.

Conclusions:

• Structured reformatting of medical guidelines and advanced prompt engineering enhance LLM efficacy in CDSSs.
• Optimizing data quality over quantity is key for effective LLM integration in healthcare.
• LLMs show promise for improving guideline delivery and clinical decision-making in hospital workflows.