Assessing Large Language Model Performance Related to Aging in Genetic Conditions

Area of Science:

• Biomedical Informatics
• Genetics
• Artificial Intelligence

Background:

• Genetic conditions are often described in pediatric populations, with limited understanding of adult manifestations and management.
• A knowledge gap exists regarding the clinical progression and lifelong care of individuals with genetic disorders.
• Generative artificial intelligence, particularly large language models (LLMs), offers potential solutions for complex biomedical data analysis.

Purpose of the Study:

• To evaluate the capability of LLMs in handling age-related aspects of 282 genetic conditions.
• To assess LLM performance in generating accurate medical vignettes and patient-geneticist dialogues.
• To identify potential age-based biases or limitations in LLM outputs for genetic disorder information.

Main Methods:

• Categorized 282 genetic conditions based on age of presentation and management changes.
• Evaluated Llama-2-70b-chat (70b) and GPT-3.5 (GPT) for generating medical vignettes, assessing correctness, completeness, and conciseness.
• Utilized generated vignettes as prompts for creating and evaluating patient-geneticist dialogues on age-based management.

Main Results:

• Both 70b and GPT demonstrated impressive performance in generating accurate medical vignettes.
• No significant overall age-based biases were observed in LLM outputs.
• Statistically significant differences were noted in specific areas, indicating nuanced LLM performance variations.

Conclusions:

• LLMs exhibit strong capabilities in processing and generating information on age-related genetic condition features.
• While promising, LLMs currently have limitations for direct clinical application in genetic medicine.
• Further development and validation are necessary to fully leverage LLMs for comprehensive genetic disorder management across the lifespan.