Enhancement of the Performance of Large Language Models in Diabetes Education through Retrieval-Augmented Generation:...

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Informatics
• Natural Language Processing

Background:

• Large language models (LLMs) show promise in processing clinical data but lack medical specificity and can generate errors.
• Patients with diabetes frequently seek health information online, highlighting the need for reliable sources.
• The Retrieval-augmented Information System for Enhancement (RISE) framework was developed to address these limitations.

Purpose of the Study:

• To evaluate the RISE framework's effectiveness in improving LLM accuracy and safety for diabetes-related patient queries.
• To assess the impact of RISE on the comprehensiveness and understandability of LLM responses.

Main Methods:

• The RISE framework involves query rewriting, information retrieval, summarization, and execution.
• Three LLMs (GPT-4, Claude 2, Bard) were tested with and without RISE using 43 diabetes questions.
• Clinicians assessed accuracy and comprehensiveness; patients evaluated understandability.

Main Results:

• RISE integration significantly boosted LLM accuracy, increasing correct responses by an average of 12%.
• Specific accuracy gains were observed across tested LLMs: GPT-4 (+7%), Claude 2 (+19%), and Bard (+9%).
• RISE also enhanced response comprehensiveness (mean score +0.44) and understandability (mean score +0.19).

Conclusions:

• The RISE framework substantially improves LLM performance for diabetes inquiries, enhancing accuracy, comprehensiveness, and understandability.
• These advancements support RISE's potential in patient education and chronic disease self-management.
• Implementing RISE can help alleviate medical resource strain and improve public health literacy.