A No-Code, Guideline-Based Custom GPT Outperforms Cardiologists in Response Quality for Cardiac Amyloidosis

Cardiac amyloidosis (CA) is increasingly recognized in clinical practice. Whether a guideline-based large language model can deliver clinician-level answer quality for CA remains unknown.

This study aimed to develop a guideline-based custom generative pretrained transformer (GPT) (AmyloGPT) and evaluate whether its response quality matches or exceeds that of board-certified cardiologists for questions regarding CA.

AmyloGPT was built in OpenAI's GPT Builder without programming, integrating the 2020 Japanese Circulation Society CA guidelines as its knowledge base. Ten nonspecialist physicians generated 71 unique clinical questions. Five board-certified cardiologist answerers drafted responses. In a prospective, blinded, comparative study, evaluators (10 nonspecialists and 3 board-certified cardiologists) assessed paired responses for preference (forced-choice) and response quality using five-point Likert scales.

Compared with cardiologist answers, AmyloGPT was preferred in 81.1% (95% CI: 78.1%-83.8%) of evaluations by nonspecialist evaluators and 83.6% (95% CI: 78.6%-88.6%) of those by cardiologist evaluators (both P < 0.001). Among nonspecialists, AmyloGPT received higher median ratings for intent alignment and clinical usefulness (both P < 0.001). Among cardiologist evaluators, AmyloGPT received higher median ratings across all 5 quality dimensions: accuracy, consistency, validity, completeness, and absence of bias (all P < 0.001).

A no-code, guideline-based custom GPT delivered superior response quality to that of cardiologists for CA questions. This approach allows clinicians without programming skills to build disease-specific large language models, potentially supporting equitable care where specialist access is limited. However, further studies are needed to evaluate potentially inaccurate outputs such as hallucinations.