Evaluation of Commonly Utilized Artificial Intelligence Large Language Models in Optimizing Readability, Accuracy,...

Online patient educational materials (PEMs) have poor readability, limiting their intended purposes in improving patient comprehension of health topics. Orthopaedic oncology PEMs are particularly complex. Although ChatGPT has demonstrated limited success in simplifying PEMs to the recommended sixth-grade reading level, other large language models (LLMs) have not been thoroughly evaluated. The goals of this study were to (1) assess baseline readability of online orthopaedic oncology PEMs, (2) evaluate five LLMs (ChatGPT-4o, Google Gemini, DeepSeek AI, Microsoft Copilot, and Meta AI) for improving readability while preserving accuracy and comprehension, and (3) to examine tradeoffs when PEMs were simplified below the sixth-grade level.

Seventy-two PEMs were collected from academic and professional sources. Readability metrics included the Flesch-Kincaid Grade Level (FKGL), Gunning Fog Index (GFI), and Flesch Reading Ease (FRE). Each PEM was rewritten by the five LLMs using the prompt: "rewrite this document to a sixth-grade reading level." Two independent graders then evaluated outputs for comprehension and accuracy (F1 score). ANOVA with pairwise comparisons assessed differences among LLMs and versus baseline (PEMs as written). A secondary analysis evaluated the effect on readability, accuracy, and comprehension of prompts to the fifth-grade, fourth-grade, and third-grade reading level.

Baseline FKGL (8.7 ± 1.5) was between the eighth-grade and ninth-grade reading level, and GFI (10.5 ± 1.9) was slightly higher. Baseline FRE was 53.9 ± 8.2. All LLMs significantly improved readability (P < 0.001), and ChatGPT-4o, DeepSeek AI, and Google Gemini conversion produced the most readable outputs. Google Gemini achieved the highest F1 score of 0.986 (range: 0.765-0.986) and 100% comprehension. Accuracy and comprehension were compromised for MetaAI when prompted below sixth grade.

ChatGPT-4o, Google Gemini, and DeepSeekAI effectively improved readability while preserving comprehension and accuracy. These findings may guide patient use of LLMs and inform healthcare-AI partnerships.