Digitalizing Medical Forms Through Visual Question Answering: Are We There Yet?

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Natural Language Processing

Background:

• Healthcare data often exists in unstructured formats, hindering efficient analysis and retrieval.
• Automating data extraction from clinical documents is crucial for improving healthcare efficiency and research.

Purpose of the Study:

• To evaluate the efficacy of multimodal neural networks in converting unstructured medical documents into structured data.
• To assess the performance of advanced Visual Question Answering models on neurological medical records.

Main Methods:

• Curated a dataset from neurological documents at the University Medical Center Hamburg-Eppendorf.
• Employed and assessed various multimodal neural network models, including recent 2024 advancements.
• Utilized Visual Question Answering techniques for data extraction.

Main Results:

• Recent models demonstrated improved performance but did not meet clinical accuracy standards.
• Significant challenges were identified in applying these technologies to complex and heterogeneous medical records.
• Current model capabilities lag behind human-level performance in clinical data extraction.

Conclusions:

• Further development is required to achieve clinically acceptable accuracy for automated data extraction.
• Larger and more diverse datasets are essential for training robust multimodal neural networks.
• The complexity of clinical data necessitates ongoing research and model refinement.