SHREC: A framework for advancing next-generation computational phenotyping with large language models

Area of Science:

• Biomedical Informatics
• Artificial Intelligence in Healthcare

Background:

• Computational phenotyping is crucial for cohort identification but is labor-intensive due to manual data review.
• Limited automation in current phenotyping methods hinders efficiency and scalability.

Purpose of the Study:

• To evaluate the efficacy of lightweight large language models (LLMs) in automating computational phenotyping tasks.
• To introduce SHREC, a framework for integrating LLMs into phenotyping pipelines.

Main Methods:

• Applied three lightweight LLMs (Gemma2, Mistral Small, Phi-4) within the SHREC framework.
• Tested LLMs for concept classification and patient phenotyping using phenotypes for Acute Respiratory Failure (ARF) respiratory support therapies.
• Evaluated model performance using Area Under the Receiver Operating Characteristic curve (AUROC) and specificity.

Main Results:

• All tested LLMs performed well on concept classification, with Mistral Small achieving an AUROC of 0.896.
• For phenotyping, LLMs demonstrated high specificity, and Mistral Small achieved an average AUROC of 0.853 for single-therapy phenotypes.
• LLMs showed adaptability to new tasks via prompt engineering and could integrate raw Electronic Health Record (EHR) data.

Conclusions:

• Lightweight LLMs show significant potential to assist researchers in resource-intensive phenotyping tasks.
• The SHREC framework facilitates the integration of LLMs for next-generation computational phenotyping.
• Future research should focus on optimizing biomedical data integration and understanding LLM reasoning errors.