From Guidelines to Real-Time Conversation: Expert-Validated Retrieval-Augmented and Fine-Tuned GPT-4 for Hepatitis C...

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing for Healthcare
• Chronic Disease Management Technologies

Background:

• Large language models (LLMs) show potential for enhancing chronic disease management, specifically for Hepatitis C Virus (HCV) infection.
• Evaluating AI techniques like retrieval-augmented generation (RAG) and supervised fine-tuning (SFT) is crucial for their clinical application.

Purpose of the Study:

• To assess the impact of RAG and SFT on LLM performance in HCV management.
• To evaluate LLM accuracy and clarity in open-ended question answering.
• To determine the effectiveness of LLMs in recommending appropriate treatment regimens for HCV clinical scenarios.

Main Methods:

• Utilized GPT-4 Turbo with baseline, RAG-Top1, RAG-Top10, and SFT configurations.
• Employed the 2020 EASL HCV guidelines for external knowledge and fine-tuning.
• Assessed LLM responses to 15 expert-vetted questions for accuracy and clarity.
• Evaluated DAA regimen recommendations in 25 simulated clinical scenarios against expert consensus.

Main Results:

• RAG-Top10 significantly improved accuracy (91.7%) and clarity (91.7%) over baseline (36.6% accuracy, 46.6% clarity) for question answering.
• SFT also enhanced performance, achieving 71.7% accuracy and 88.3% clarity.
• RAG-Top10 achieved the highest performance in recommending correct DAA regimens (76% vs. 24% for baseline).

Conclusions:

• Both RAG and SFT substantially enhance LLM capabilities for guideline-driven HCV management.
• RAG-Top10 offers the greatest benefits due to broader context retrieval.
• Domain-specific alignment through SFT is valuable, highlighting the need for expert-informed evaluation frameworks for safe LLM integration.