Assessing large language model performance related to aging in genetic conditions

Area of Science:

• Medical Genetics
• Artificial Intelligence in Medicine
• Clinical Informatics

Background:

• Genetic conditions are often described in pediatric populations, creating a knowledge gap for adult patient care.
• Large language models (LLMs) show promise in various applications, prompting investigation into their medical utility.

Purpose of the Study:

• To evaluate the ability of Llama-2-70b-chat (70b) and GPT-3.5 (GPT) to generate age-appropriate medical vignettes, dialogues, and management plans for genetic conditions in both child and adult hypothetical patients.
• To assess the correctness and completeness of LLM-generated content for 282 genetic conditions.

Main Methods:

• LLMs (70b and GPT) were prompted to create medical vignettes, patient-geneticist dialogues, and management plans for hypothetical pediatric and adult patients across 282 genetic conditions.
• Clinicians graded the generated content for correctness and completeness, focusing on age-appropriateness and clinical plausibility.
• A sub-analysis was performed on metabolic conditions, including those presenting neonatally.

Main Results:

• LLMs demonstrated age-appropriate responses in generating medical vignettes and dialogues for both child and adult patients, as indicated by clinician scoring.
• Sub-analysis confirmed age-appropriate LLM responses for metabolic conditions, even those with neonatal crisis presentations.
• Both 70b and GPT models achieved lower correctness and completeness scores for generating plausible clinical management plans, ranging from 50-90% depending on the model and specific metrics.

Conclusions:

• LLMs show potential in generating age-specific medical information for genetic conditions, aiding in bridging pediatric-adult knowledge gaps.
• Current LLMs have limitations in producing clinically accurate and complete management plans, highlighting areas for future development and validation in healthcare applications.