Evaluation of artificial language-adjusted readability on cataract surgery queries

To analyze accuracy and readability of answers to cataract surgery queries produced by artificial intelligence (AI) models and determine whether AI models can significantly improve readability.

Google Gemini Advanced, ChatGPT 4.0, and Microsoft Copilot Pro were prompted to answer 25 questions about cataract surgery, followed by a request to re-answer questions at a 6th-grade level. Objective readability of answers were measured with five validated reading formulas and word count. Accuracy and readability of each answer were further graded by three ophthalmologists. Comparisons were performed between original and 6th-grade versions and among the three AI models.

After being prompted to answer at a 6th-grade reading level, Google Gemini Advanced and Microsoft Copilot Pro had lower average reading level than ChatGPT 4.0 (8.04 vs 8.19 vs 9.43 [P < 0.001]). Microsoft Copilot answers had higher Flesch reading ease score (75.40 vs 71.24 vs 69.46 [P < 0.007]) and lower word count (130.28 vs 180.24 vs 166.08 [P < 0.001]) among AI models. Microsoft Copilot Pro and ChatGPT 4.0 answers had greater change in reading level (-6.13 vs -5.75 vs -3.31 [P < 0.001]) and Flesch reading ease score (39.67 vs 35.98 vs 23.67 [P < 0.001]) compared with Google Gemini Advanced. Graders determined that there were no changes in accuracy before and after being prompted to answer at a 6th-grade reading level.

AI models can simplify reading level of responses to common cataract surgery queries while maintaining accuracy.