Extracting Clinical Guideline Information Using Two Large Language Models: Evaluation Study

Area of Science:

• Pharmacogenomics (PGx)
• Artificial Intelligence (AI)
• Clinical Decision Support Systems (CDSS)

Background:

• Effective personalized pharmacogenomics (PGx) requires integrating clinical guidelines into decision support systems.
• Large language models (LLMs) offer potential for automating the extraction and updating of PGx information.
• Manual review of PGx guidelines is time-consuming and resource-intensive.

Purpose of the Study:

• To assess the effectiveness of repeated cross-comparisons and an agreement-threshold strategy using two advanced LLMs for updating PGx clinical guidelines.
• To evaluate the performance of GPT-4o and Gemini-1.5-Pro in extracting and classifying PGx guidelines.
• To determine the potential of LLMs to streamline the integration of PGx guidelines into clinical practice.

Main Methods:

• Two LLMs (GPT-4o, Gemini-1.5-Pro) classified 385 PGx clinical guidelines, with each tested 20 times per model.
• Strategies included repeated cross-comparison and a consistency threshold (predictions <60% agreement) to flag inconsistencies.
• LLM outputs were compared against expert-annotated data for accuracy assessment.

Main Results:

• High reproducibility rates were achieved by both LLMs (GPT-4o: 97.8%, Gemini-1.5-Pro: 98.9%).
• LLMs demonstrated high accuracy (GPT-4o: 93.5%, Gemini-1.5-Pro: 92.7%) compared to expert labels.
• Consistent predictions reduced manual review needs by 88.6%, with minimal error rates (0.3-0.5%) and very low cost (US $0.76).

Conclusions:

• Utilizing two LLMs offers a cost-effective and scalable method for updating PGx guidelines for clinical decision support.
• Automated classification by LLMs significantly reduces the burden of manual review, enhancing clinical applicability.
• Selective manual review remains crucial for ensuring accuracy, but this LLM-driven approach optimizes PGx guideline integration.