Enhancing Large Language Model Reliability: Minimizing Hallucinations with Dual Retrieval-Augmented Generation Based...

Area of Science:

• Artificial Intelligence in Medicine
• Medical Informatics
• Natural Language Processing

Background:

• Large language models (LLMs) show potential in healthcare but struggle with accuracy (hallucinations), especially in dynamic fields like diabetes management.
• Existing LLM updating methods are resource-intensive, creating a need for efficient ways to provide current medical information.
• Ensuring the reliability and up-to-dateness of AI-generated medical content is critical for safe clinical application.

Purpose of the Study:

• To develop and evaluate a novel retrieval system to enhance the reliability of large language models (LLMs) in diabetes management.
• To create a dual retrieval-augmented generation (RAG) system capable of integrating and utilizing information from diverse international guidelines.
• To assess the system's performance across different languages and identify optimal retrieval strategies for improved accuracy.

Main Methods:

• A dual retrieval-augmented generation (RAG) system was developed, integrating the Korean Diabetes Association and American Diabetes Association 2023 guidelines.
• The system utilized dense retrieval with 11 embedding models (including OpenAI, Upstage, and multilingual options) and sparse retrieval via the BM25 algorithm with language-specific tokenizers.
• Performance was evaluated using various top-k values to optimize ensemble retrievers for each guideline, focusing on both Korean and English texts.

Main Results:

• For dense retrieval, Upstage's Solar Embedding-1-large and OpenAI's text-embedding-3-large excelled for Korean and English, respectively; multilingual models outperformed language-specific ones.
• The ko_kiwi tokenizer showed superior performance for Korean sparse retrieval, while ko_kiwi and porter_stemmer were comparable for English.
• Optimized ensemble retrievers, combining dense and sparse methods, improved information coverage while maintaining precision.

Conclusions:

• A dual RAG system effectively enhances LLM reliability for diabetes management information across languages.
• The system's successful application with both Korean and American guidelines demonstrates its cross-regional utility.
• This work provides a foundation for developing more trustworthy AI-assisted healthcare applications in diverse global contexts.