Evaluating Large Language Models for Translating Multimodal Phenotype Documentations into Executable EHR Phenotyping...

Area of Science:

• Health Informatics
• Artificial Intelligence in Medicine
• Clinical Data Management

Background:

• Translating clinical definitions into executable electronic health record (EHR) database queries is crucial for research but is a labor-intensive process.
• The increasing availability of EHR data necessitates efficient methods for phenotype extraction and query generation.

Purpose of the Study:

• To evaluate the performance of two advanced large language models (LLMs) in generating EHR database queries from clinical definitions.
• To assess the impact of different documentation modalities (e.g., structured text, diagrams) on LLM performance for EHR phenotype translation.

Main Methods:

• Two state-of-the-art large language models were tested on five distinct clinical phenotypes.
• The models processed three types of documentation: structured text, semi-structured text, and diagrams.
• Performance was evaluated based on the accuracy and completeness of the generated EHR database queries.

Main Results:

• Both evaluated LLMs demonstrated proficiency in capturing high-level logic from structured and semi-structured clinical documentation.
• Model performance significantly degraded when presented with diagram-only input, indicating limitations in interpreting visual data.
• Error analysis identified seven distinct categories of failures, with documentation quality emerging as the primary bottleneck.

Conclusions:

• While LLMs show potential for automating EHR phenotype query generation, current capabilities are constrained by input documentation quality.
• Standardization of clinical documentation and continued expert oversight are essential to overcome current limitations and improve LLM utility in EHR research.
• Future research should focus on improving LLM's ability to interpret diverse documentation formats and addressing the impact of data quality.