Improving Retrieval Augmented Generation for Health Care by Fine-Tuning Clinical Embedding Models: Development and...

Area of Science:

• Medical Informatics
• Natural Language Processing
• Artificial Intelligence in Healthcare

Background:

• Existing embedding models for Retrieval Augmented Generation (RAG) are primarily trained on English data, limiting their use in non-English healthcare.
• These models often lack training on real-world clinical documents, leading to inaccurate context retrieval in specialized medical settings.
• Domain-specific terminology, abbreviations, and nuanced language in clinical documents pose challenges for general embedding models.

Purpose of the Study:

• To develop and validate embedding models specifically trained on real-world clinical documents.
• To improve medical information retrieval (IR) and RAG system performance in both German and English contexts.
• To address limitations of general embedding models in specialized healthcare documentation.

Main Methods:

• Fine-tuned sentence transformers using the multilingual-e5-large architecture.
• Generated ~11 million synthetic question-answer pairs from 400,000 clinical documents.
• Utilized SauerkrautLM-SOLAR-Instruct LLM for question-answer generation and translated data to English.

Main Results:

• The fine-tuned model achieved a mAP@100 of 0.27 in IR tasks, outperforming baselines (multilingual-e5-large: 0.14, bge-m3: 0.11).
• Demonstrated robust RAG performance, comparable to baselines in patient-centered scenarios and moderate improvements in cross-patient settings.
• Models trained on pseudonymized data showed strong retrieval performance and high contextual precision.

Conclusions:

• Developed and validated domain-specific embedding models using real-world clinical data and LLM-generated synthetic data.
• These models enhance medical IR and RAG applications, particularly in specialized healthcare contexts.
• Published models offer a reproducible framework for improving medical data retrieval in diverse healthcare institutions.