Zero-shot learning for clinical phenotyping: Comparing LLMs and rule-based methods

Phenotyping, the process of systematically identifying and classifying conditions within clinical data, is a crucial first step in any data science work involving Electronic Health Records (EHRs). Traditional approaches require extensive manual annotation efforts and face challenges with scalability.

We investigated the use of Large Language Models (LLMs) for zero-shot phenotyping of 20 prevalent chronic conditions based on synthetic patient summaries generated from real structured EHRs codes. We evaluated the performance of multiple LLMs, including GPT-4o, GPT-3.5, and LLaMA 3 models with 8-billion, 70-billion, and 405-billion parameters, comparing them against traditional rule-based methods. For the analysis we used a dataset of 1,000 patients from Hospital da Luz Lisboa.

GPT-4o outperformed both traditional rule-based methods and alternative LLMs, achieving superior recall (0.97) and macro-F1 score (0.92). Rule-based phenotyping, while highly precise (0.92), showed lower recall (0.36). The integration of rule-based methods with LLMs optimized phenotyping accuracy by targeting manual annotation efforts on discordant cases.

Zero-shot learning with LLMs, particularly GPT-4o, offers a powerful and efficient approach for phenotyping chronic conditions from EHRs, significantly reducing the need for extensive labeled datasets while maintaining high accuracy and interpretability.