Optimizing Data Extraction: Harnessing RAG and LLMs for German Medical Documents.

Area of Science:

• Medical Informatics
• Natural Language Processing
• Data Privacy

Background:

• Converting unstructured medical documents into structured data is a significant challenge in medical data analysis.
• Sensitive health information requires secure and efficient processing methods.

Purpose of the Study:

• To develop and evaluate an automated, locally deployed, data-privacy-secure pipeline for structuring German medical documents.
• To leverage open-source Large Language Models (LLMs) with Retrieval-Augmented Generation (RAG) for this task.

Main Methods:

• Implementation of a secure pipeline using open-source LLMs and RAG architecture.
• Deployment on a local infrastructure ensuring data privacy.
• Testing on a proprietary dataset of 800 unstructured German medical reports.

Main Results:

• The pipeline demonstrated high accuracy, achieving up to 90% in data extraction.
• Performance was validated against manual extraction by physicians and medical students.
• Successful conversion of sensitive health-related information into a structured format.

Conclusions:

• The developed pipeline offers a valuable tool for efficient data extraction from unstructured medical sources.
• The LLM-RAG approach provides a privacy-secure and accurate solution for medical data analysis.
• This technology has the potential to streamline medical data processing and research.