Readability and Citation Sources of ChatGPT-Generated Responses to Lymphoma-Related Health Questions: Cross-Sectional...

Lymphoma is one of the most prevalent groups of cancers among people aged 15 to 29 years. Increasingly, patients with lymphoma and their caregivers use generative AI systems, supported by large language models (LLMs), to seek cancer-related online health information. Despite the need for plain and clear health information for clinical decision-making, there is a dearth of research on the readability and citation source classification of LLM-supported online health information.

The objective of this study was to assess the reading level and reading ease of lymphoma-related health information using OpenAI's GPT-5.2 model, as well as the citation sourcing of LLM-generated results.

Questions were submitted to the LLM through an automated pipeline. The application code was written with Claude and run and hosted on Vercel. Standardized instructions were submitted to the LLM, asking for evidence-based sources with URLs to enable citation analysis. The final dataset included 392 citations garnered from 96 standardized lymphoma-related questions. Readability was assessed using the Flesch-Kincaid Grade Level and Flesch Reading Ease scores. Citation sources were classified into 9 predefined organization types, and topics included Hodgkin and non-Hodgkin lymphoma and clinical trials. Differences in readability were tested using linear models fitted to all 392 citations, with standard errors clustered by question. Estimated marginal means (EMMs) were used to summarize model-adjusted readability scores by citation source and topic. When the omnibus test was statistically significant, pairwise comparisons were performed using the Tukey-Kramer adjustment for multiple comparisons.

The Flesch-Kincaid reading levels of the ChatGPT responses by citation source and topic type were determined to be at the college or university level, well above the recommended sixth-grade level. Grade level did not differ by citation source (P=.58). The reading grade level of responses for Hodgkin lymphoma responses was statistically higher than that of clinical trial responses (mean 16.2, SD 3.4 vs 13.6, SD 2.5; P=.005). Flesch Reading Ease scores across source and topic were found to be very difficult at the university graduate academic level and did not differ by source (P=.27) or topic (P=.17).

Although LLMs frequently produce responses that include cited information, the readability of these outputs may limit accessibility for many patients and caregivers. These findings highlight the need for improved readability standards in AI-generated health information and reinforce that LLM outputs should complement, not replace, professional medical guidance.