A No-Code, Guideline-Based Custom GPT Outperforms Cardiologists in Response Quality for Cardiac Amyloidosis

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Medical Guideline Implementation

Background:

• Cardiac amyloidosis (CA) is increasingly recognized in clinical practice.
• The potential of guideline-based large language models (LLMs) to achieve clinician-level answer quality for CA is currently unknown.

Purpose of the Study:

• To develop a custom generative pretrained transformer (GPT) model, AmyloGPT, based on clinical guidelines.
• To evaluate if AmyloGPT's response quality for CA matches or surpasses that of board-certified cardiologists.

Main Methods:

• AmyloGPT was developed using OpenAI's GPT Builder, integrating the 2020 Japanese Circulation Society CA guidelines.
• Seventy-one clinical questions were generated by nonspecialist physicians and answered by board-certified cardiologists.
• A blinded, comparative study assessed paired responses by nonspecialist and cardiologist evaluators using preference and Likert scales.

Main Results:

• AmyloGPT was preferred over cardiologist responses by 81.1% of nonspecialist evaluators and 83.6% of cardiologist evaluators (P < 0.001).
• Nonspecialist evaluators rated AmyloGPT higher for intent alignment and clinical usefulness (P < 0.001).
• Cardiologist evaluators rated AmyloGPT higher across all quality dimensions: accuracy, consistency, validity, completeness, and absence of bias (P < 0.001).

Conclusions:

• A no-code, guideline-based custom GPT (AmyloGPT) demonstrated superior response quality compared to cardiologists for CA-related queries.
• This approach enables clinicians to create disease-specific LLMs without programming, potentially enhancing care accessibility.
• Further research is necessary to address potential inaccuracies like hallucinations in LLM outputs.