A Comparative Readability Analysis of Artificial Intelligence-Generated and Evidence-based Medical Content on the...

Multiple sclerosis (MS) is a complex neurological disorder requiring effective patient education. With the increasing use of artificial intelligence (AI) in healthcare, evaluating the readability of AI-generated content compared to evidence-based resources is essential to ensure patient accessibility.

This study aims to compare the readability of patient education guides on the diagnosis and management of MS generated by the AI tool Google Gemini with a standard clinical reference, UpToDate.

Content was analyzed using the metrics including word count, sentence count, difficult word count/percentage, flesch reading ease (FRE), Flesch-Kincaid Grade Level (FKGL), and simple measure of gobbledygook (SMOG) index. The Wilcoxon signed-rank test was used for statistical comparison.

Statistically significant differences (P < 0.05) were found for word count, sentence count, and difficult word count. UpToDate had a significantly higher median word count (6332.5 vs. 800.5) and sentence count (199.5 vs. 44.0) than Google Gemini, indicating it was more verbose and complex. Google Gemini produced content that was relatively easier to read based on FRE, FKGL, and SMOG Index, although these differences were not statistically significant (P > 0.05). Importantly, neither source met the recommended 6th to 8th-grade readability levels for patient health materials.

The overall readability scores (FRE, FKGL, and SMOG) were similar, and neither platform delivers information at the comprehension level recommended for the average patient. AI tools such as Google Gemini may serve as a useful adjunct for brief, patient-oriented information, but further refinement is needed to improve accessibility.