Towards automated phenotype definition extraction using large language models.

Area of Science:

• Biomedical Informatics
• Computational Health
• Artificial Intelligence in Medicine

Background:

• Electronic phenotyping analyzes diverse data for health insights.
• Current phenotype definition extraction is manual, time-consuming, and unscalable.
• Large language models (LLMs) offer automation but face reliability challenges like hallucinations.

Purpose of the Study:

• To establish a standard evaluation set for LLM-generated phenotype definitions.
• To assess various prompting strategies for extracting phenotype definitions using LLMs.
• To improve the reliability and utility of LLM outputs in phenotype extraction.

Main Methods:

• Development of a standardized evaluation dataset for phenotype definitions.
• Implementation and testing of diverse prompting techniques for LLM-based extraction.
• Assessment of LLM-generated definitions against the established evaluation task.

Main Results:

• Promising results were achieved in extracting phenotype definitions using LLMs.
• The developed evaluation set aids in assessing the reliability of LLM outputs.
• Prompting strategies show potential for enhancing phenotype extraction efficiency.

Conclusions:

• LLM-based phenotype extraction shows potential to reduce manual review time.
• Human evaluation and validation remain crucial for ensuring accuracy and safety.
• Further research can optimize LLM prompting for reliable clinical applications.