Accuracy, Usefulness, and Expert-Perceived Clinical Impact of Response Variability in ChatGPT-4o for...

ChatGPT-4o, an advanced Artificial Intelligence based large language model, has demonstrated utility for providing medical information. Data regarding its accuracy and usefulness for hyperlipidemia pharmacotherapy are limited. Additionally, the clinical implications of variability in responses generated from simultaneous identical prompts across multiple sessions remains poorly understood. This research aimed to (1) assess accuracy, (2) evaluate usefulness, and (3) examine perceived impact variability of ChatGPT-4o generated responses to hyperlipidemia-related clinical questions. Five questions addressing hyperlipidemia management and patient counseling were developed. Responses to the questions were collected simultaneously from three independent ChatGPT-4o sessions initiated in separate, newly opened ChatGPT windows. A modified Delphi technique using clinical pharmacy faculty evaluated responses for accuracy, usefulness, and impact variability. Scores were evaluated using a 3-point Likert scale. After initial scoring, responses were collected, collated, and redistributed to faculty for review and optional revision. Final evaluations and consensus were facilitated by a moderator during a virtual panel. Consensus was reached for all responses. Evaluations of response accuracy ranged from poor to good, while usefulness ranged between somewhat useful and very useful. Impact variability ranged from none to high across responses, which faculty concluded could impact patient management. Response variability may have potential clinical implications; however, the present study evaluated expert-perceived potential impact rather than actual patient outcomes. Users should be aware concurrently generated ChatGPT responses can differ in accuracy and usefulness. Clinical judgment and patient-centered multidisciplinary collaboration remain essential. Future directions include using a larger data set to corroborate findings.