Generative artificial intelligence for automated data extraction from unstructured medical text.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Data Extraction

Background:

• Unstructured clinical notes, like right heart catheterization (RHC) procedure notes, contain valuable data but are underutilized due to manual extraction challenges.
• Automating data extraction from these notes is crucial for improving research efficiency and clinical applications.

Purpose of the Study:

• To develop and validate a generative artificial intelligence (GenAI) pipeline for automated data extraction from unstructured RHC notes.
• To minimize errors, including hallucinations, during data extraction using a Large Language Model (LLM) with built-in guardrails and a retry mechanism.

Main Methods:

• A GenAI pipeline was developed using an open-source LLM, incorporating an Engineered Preload Framework (EPF) with schemas and instructions.
• The pipeline included an LLM module with reasoning capabilities and a validation/retry mechanism for self-correction.
• Performance was evaluated against manually extracted ground truth from 200 RHC notes using precision, recall, and F1 score.

Main Results:

• The GenAI pipeline achieved 99.0% precision, 85.0% recall, and a 91.5% F1 score, with 90% overall note-level accuracy.
• Missed values were the most common error (5.2%), while hallucinations were minimal (<0.01%).
• The pipeline demonstrated robust performance across varying data availability levels.

Conclusions:

• A feasible and robust GenAI pipeline for automated structured data extraction from unstructured RHC notes was demonstrated.
• This approach highlights the potential of LLMs for efficient medical data mining, enhancing research and clinical practice.