Development and Evaluation of a Retrieval-Augmented Generation-Based Electronic Medical Record Chatbot System

Area of Science:

• Health Informatics
• Artificial Intelligence in Healthcare
• Natural Language Processing

Background:

• Hospital operations face challenges with administrative efficiency and accessing information from complex electronic medical record (EMR) manuals.
• Streamlining workflows and enhancing healthcare delivery are critical for optimizing hospital performance.

Purpose of the Study:

• To develop and evaluate a retrieval-augmented generation (RAG)-based chatbot system for optimizing hospital operations.
• To leverage EMR manuals for improved administrative workflows and healthcare delivery.

Main Methods:

• Integrated fine-tuned multilingual embedding models (Multilingual-E5-Large, BGE-M3) for indexing and retrieval from EMR manuals.
• Constructed a dataset of 5,931 question-document pairs with expert validation.
• Utilized contrastive learning for fine-tuning and the Solar Mini Chat API for Korean-language text generation.

Main Results:

• Fine-tuned models showed significant retrieval accuracy improvements (BGE-M3: 97.6%, Multilingual-E5-Large: 89.7%).
• The chatbot demonstrated low query latency (<10 ms) and high retrieval precision for operational EMR queries.
• Applications included administrative task support and billing process optimization, reducing staff workload.

Conclusions:

• The RAG chatbot effectively improved EMR usability and operational efficiency in healthcare administration.
• Further real-world deployment and longitudinal studies are recommended to assess administrative burden reduction and workflow improvements.