Benchmarking large language models for replication of guideline-based PGx recommendations

Area of Science:

• Artificial Intelligence in Medicine
• Pharmacogenomics
• Clinical Decision Support

Background:

• Large language models (LLMs) are increasingly explored for clinical applications.
• Accurate pharmacogenomic (PGx) recommendations are crucial for personalized medicine.
• Existing LLMs may lack the specificity for complex clinical guidelines like CPIC.

Purpose of the Study:

• To evaluate the clinical accuracy of LLMs in generating pharmacogenomic (PGx) recommendations.
• To compare the performance of general-purpose LLMs against domain-adapted models for PGx.
• To establish a framework for assessing PGx-specific LLM performance.

Main Methods:

• A benchmark of 599 curated gene-drug-phenotype scenarios was used.
• Five leading LLMs, including GPT-4o and fine-tuned LLaMA variants, were evaluated.
• A novel semantic evaluation framework (LLM Score), validated by expert review, was employed alongside lexical metrics.

Main Results:

• General-purpose LLMs frequently generated incomplete or unsafe PGx recommendations.
• A domain-adapted LLM achieved a high LLM Score of 0.92, indicating superior performance.
• The domain-adapted model demonstrated significantly faster inference speeds compared to others.

Conclusions:

• Fine-tuning and structured prompting are critical for developing accurate PGx LLMs, surpassing model scale alone.
• Domain-specific adaptation is essential for safe and effective AI in pharmacogenomics.
• This study validates a framework for evaluating PGx LLMs and supports AI-driven personalized medicine.