Accuracy of an artificial intelligence chatbot in identifying congenital glaucoma from other ocular etiologies

To evaluate the accuracy and appropriateness of responses provided by ChatGPT in identifying congenital glaucoma from a series of written and image-based prompts.

A series of questions regarding common signs and symptoms of congenital glaucoma were developed and queried to ChatGPT-3.5 and ChatGPT-4.0, and a set of publicly available images of patients with congenital glaucoma were queried to the image search function of ChatGPT-4.0. Outputs were graded by three pediatric ophthalmologists with expertise in congenital glaucoma. Completeness of response, accuracy, potential for harm, and concern for glaucoma were assessed by each reviewer.

A higher proportion of prompt responses from ChatGPT-4.0 were graded to be acceptable/appropriate than from ChatGPT-3.5 (22/33 vs 9/33 [P = 0.001]) among text-based queries. A higher proportion of ChatGPT-4.0 responses were felt to raise appropriate concern for congenital glaucoma (8/11 vs 2/11 [P = 0.03]) and a lower proportion of responses had incorrect or inappropriate information of major clinical significance (0/33 vs 6/33 [P = 0.02]) than ChatGPT-3.5 responses. There was no significant difference in the proportion of responses from ChatGPT-3.5 and ChatGPT-4.0 that were deemed to have potential likelihood of harm (P = 0.17). Among clinical images queried to ChatGPT-4.0, responses to two of three images were universally felt to be unacceptable with a major amount of incorrect or inappropriate clinical information and high/definitive likelihood of harm. Among readability indices, the SMOG Index score showed more difficult readability scores for ChatGPT-4.0 than for ChatGPT-3.5 (14.8 ± 1.2 vs 14.0 ± 1.4 [P = 0.009]).

Despite superior performance from ChatGPT-4.0 compared with ChatGPT-3.5 in raising concern for congenital glaucoma and appropriateness of responses from text-based prompts, it performed poorly in recognizing clinical images of congenital glaucoma.