Towards automated phenotype definition extraction using large language models

Ramya Tekumalla ¹, Juan M Banda ^2,3

⁰Mercer University, Atlanta, GA, USA.

Genomics & Informatics +

|

November 1, 2024

View abstract on PubMed

Summary

Large language models can automate phenotype definition extraction, but require careful evaluation. This study developed a standard evaluation set and tested prompting methods to improve reliability in electronic phenotyping.

Area Of Science

Biomedical Informatics
Computational Health
Artificial Intelligence in Medicine

Background

Electronic phenotyping analyzes diverse data for health insights.
Current phenotype definition extraction is manual, time-consuming, and unscalable.
Large language models (LLMs) offer automation but face reliability challenges like hallucinations.

Purpose Of The Study

To establish a standard evaluation set for LLM-generated phenotype definitions.
To assess various prompting strategies for extracting phenotype definitions using LLMs.
To improve the reliability and utility of LLM outputs in phenotype extraction.

Main Methods

Development of a standardized evaluation dataset for phenotype definitions.
Implementation and testing of diverse prompting techniques for LLM-based extraction.
Assessment of LLM-generated definitions against the established evaluation task.

Main Results

Promising results were achieved in extracting phenotype definitions using LLMs.
The developed evaluation set aids in assessing the reliability of LLM outputs.
Prompting strategies show potential for enhancing phenotype extraction efficiency.

Conclusions

LLM-based phenotype extraction shows potential to reduce manual review time.
Human evaluation and validation remain crucial for ensuring accuracy and safety.
Further research can optimize LLM prompting for reliable clinical applications.

Keywords:

ChatGPT Electronic phenotyping Evaluation Large language models (LLMs)