Zero-shot learning for clinical phenotyping: Comparing LLMs and rule-based methods.

Area of Science:

• Computational Health Informatics
• Artificial Intelligence in Medicine
• Data Science for Healthcare

Background:

• Phenotyping, classifying conditions in clinical data, is essential for Electronic Health Records (EHRs) data science.
• Traditional phenotyping methods are labor-intensive and difficult to scale.
• Automating phenotyping is critical for leveraging large EHR datasets.

Purpose of the Study:

• To evaluate Large Language Models (LLMs) for zero-shot phenotyping of chronic conditions.
• To compare LLM performance against traditional rule-based methods.
• To assess the efficiency and accuracy of LLM-based phenotyping in EHR data.

Main Methods:

• Investigated zero-shot phenotyping for 20 chronic conditions using synthetic patient summaries from EHRs.
• Evaluated multiple LLMs (GPT-4o, GPT-3.5, LLaMA 3) and rule-based methods.
• Utilized a dataset of 1,000 patients from Hospital da Luz Lisboa for analysis.

Main Results:

• GPT-4o achieved the highest recall (0.97) and macro-F1 score (0.92), outperforming other LLMs and rule-based approaches.
• Rule-based methods showed high precision (0.92) but lower recall (0.36).
• Integrating rule-based methods with LLMs improved overall phenotyping accuracy by focusing manual efforts.

Conclusions:

• Zero-shot learning with LLMs, especially GPT-4o, provides an efficient and accurate method for EHR phenotyping.
• LLMs significantly reduce the requirement for extensive labeled datasets.
• This approach enhances accuracy and interpretability in chronic condition phenotyping.