Artificial Doctors: Performance of Chatbots as a Tool for Patient Education on Keratoconus

We aimed to compare the answers given by ChatGPT, Bard, and Copilot and that obtained from the American Academy of Ophthalmology (AAO) website to patient-written questions related to keratoconus in terms of accuracy, understandability, actionability, and readability to find out whether chatbots can be used in patient education.

Twenty patient-written questions obtained from the AAO website related to keratoconus were asked to ChatGPT, Bard, and Copilot. Two ophthalmologists independently assessed the answers obtained from chatbots and the AAO website in terms of accuracy, understandability, and actionability according to the Structure of Observed Learning Outcome taxonomy, Patient Education Materials Assessment Tool-Understandability, and Patient Education Materials Assessment Tool-Actionability tests, respectively. The answers were also compared for readability according to the Flesch Reading Ease scores obtained through the website.

Bard had significantly higher scores compared with ChatGPT-3.5, Copilot, and AAO website according to Structure of Observed Learning Outcome taxonomy and Patient Education Materials Assessment Tool-Understandability ( P <0.001 for each), whereas there was no significant difference between the other groups. Bard and ChatGPT achieved significantly higher scores than the AAO website according to the Patient Education Materials Assessment Tool-Actionability scale ( P =0.001). The AAO website achieved significantly higher scores than the Bard on the Flesch Reading Ease scale, whereas there was no significant difference between the other groups ( P =0.017).

Chatbots are promising to provide accurate, understandable, and actionable answers. Chatbots can be a valuable aid in the education of patients with keratoconus under clinician supervision. In this way, unnecessary hospital visits can be prevented, and the burden on the health care system can be alleviated, while patient awareness can be raised.