Evaluating Large Language Models for Translating Multimodal Phenotype Documentations into Executable EHR Phenotyping...

Area of Science:

• Health Informatics
• Artificial Intelligence in Medicine
• Clinical Data Management

Background:

• Electronic Health Record (EHR) phenotypes are crucial for research.
• Translating clinical definitions into EHR database queries is a complex and time-consuming task.
• Large language models (LLMs) offer potential solutions for automating this process.

Purpose of the Study:

• To evaluate the performance of two advanced large language models in translating clinical definitions into executable EHR database queries.
• To assess the impact of different documentation modalities (structured text, diagrams) on LLM performance.
• To identify failure categories and bottlenecks in the automated query generation process.

Main Methods:

• Two frontier large language models were tested.
• Five distinct clinical phenotypes were used for evaluation.
• Three different documentation modalities were employed as input for the models.
• A detailed error analysis was conducted to categorize model failures.

Main Results:

• Both evaluated LLMs demonstrated capability in capturing high-level logic from structured text-based documentation.
• Model performance significantly degraded when presented with diagram-only input.
• Seven distinct categories of errors were identified during the analysis.
• The quality and format of documentation emerged as the primary limitation, not the LLM's core capability.

Conclusions:

• While LLMs show potential for EHR phenotype query generation, their current effectiveness is limited by input modality.
• Documentation standardization and the inclusion of expert oversight are critical for successful implementation.
• Future research should focus on improving LLM performance with visual data and standardizing clinical documentation practices.