A Multimodal Pipeline for Clinical Data Extraction: Applying Vision-Language Models to Scans of Transfusion Reaction...

Area of Science:

• Health Informatics
• Computer Vision
• Natural Language Processing

Background:

• Healthcare systems face challenges with data digitization due to reliance on paper documents.
• Manual data transcription from paper to digital formats is inefficient and error-prone.
• Heterogeneous real-world healthcare data conditions necessitate robust digital solutions.

Purpose of the Study:

• To develop and present an open-source pipeline for automated extraction and categorization of checkbox data from scanned documents.
• To streamline the workflow of transferring paper-based health data to digital formats.
• To demonstrate the pipeline's adaptability to various checkbox-rich document types, starting with transfusion reaction reports.

Main Methods:

• The pipeline integrates checkbox detection algorithms.
• Multilingual optical character recognition (OCR) is employed for text recognition.
• Multilingual vision-language models (VLMs) are utilized for advanced data interpretation.

Main Results:

• The pipeline demonstrated high precision and recall in extracting checkbox data.
• Performance was validated against gold-standard datasets compiled annually from 2017 to 2024.
• Significant reduction in administrative workload and improved accuracy in regulatory reporting were achieved.

Conclusions:

• The developed open-source pipeline offers an efficient and accurate solution for digitizing checkbox data from scanned healthcare documents.
• The system's modular design supports adaptation to diverse document types beyond transfusion reaction reports.
• Open-source availability promotes self-hosted data parsing, enhancing data management and compliance in healthcare.