SHREC: A Framework for Advancing Next-Generation Computational Phenotyping with Large Language Models.

Area of Science:

• Biomedical Informatics
• Artificial Intelligence in Healthcare

Background:

• Computational phenotyping is crucial for cohort identification but is labor-intensive due to manual data review.
• Limited automation in current phenotyping methods hinders scalability and efficiency.

Purpose of the Study:

• To evaluate the effectiveness of lightweight large language models (LLMs) in automating computational phenotyping tasks.
• To introduce SHREC, a framework for integrating LLMs into end-to-end phenotyping pipelines.

Main Methods:

• Tested three lightweight LLMs (Gemma2, Mistral Small, Phi-4) for concept classification and patient phenotyping.
• Utilized phenotypes for Acute Respiratory Failure (ARF) respiratory support therapies.
• Assessed model performance using Area Under the Receiver Operating Characteristic Curve (AUROC) and specificity.

Main Results:

• All tested LLMs performed well in concept classification, with Mistral Small achieving an AUROC of 0.896.
• Models exhibited high specificity for all phenotypes.
• The top-performing model, Mistral Small, achieved an average AUROC of 0.853 for single-therapy phenotypes.

Conclusions:

• Lightweight LLMs can effectively assist researchers in resource-intensive phenotyping tasks.
• LLMs offer advantages such as adaptability via prompt engineering and the ability to process raw Electronic Health Record (EHR) data.
• Future research should focus on optimizing biomedical data integration and understanding LLM reasoning errors.