Reliability and readability of five large language models in answering patient-oriented questions about rectal prolapse: a comparative study
Related Experiment Videos
Le Zhang1, Lei Liang2, Xin Peng2
1Department of Colorectal Surgery, Jintang Hospital Jintang First People's Hospital, West China Hospital Sichuan University, Chengdu, Sichuan, China.
Background:
Rectal prolapse (RP) is a pelvic floor disorder that can cause rectal protrusion, defecatory dysfunction, and fecal incontinence. Patients increasingly use large language models (LLMs) for health information, yet the reliability and readability of LLM responses to patient-oriented questions about RP remain unclear.
Objective:
To compare the reliability and readability of five LLM platform-model configurations in answering English-language patient-oriented questions about RP.
Methods:
In this web-based cross-sectional study, worldwide Google Trends queries from the preceding 5 years were identified using the MeSH term "rectal prolapse." Two colorectal specialists screened the queries and standardized them into 15 patient-oriented questions. The questions were submitted to ChatGPT, Gemini, Microsoft Copilot, Perplexity, and DeepSeek under standardized single-turn, zero-shot conditions, with one response collected per question from each platform-model configuration. Two independent evaluators assessed the responses using DISCERN, the Ensuring Quality Information for Patients (EQIP) instrument, the Global Quality Score (GQS), and the JAMA benchmark criteria. Using evidence-based guidelines and question-specific checklists of key clinical content, the evaluators also independently assessed clinical factual accuracy and completeness and identified potentially harmful recommendations and critical omissions. Readability was evaluated using the Automated Readability Index (ARI), Gunning Fog Index (GFI), Flesch-Kincaid Grade Level (FKGL), Coleman-Liau Index (CL), Simple Measure of Gobbledygook (SMOG), and Flesch Reading Ease Score (FRES).
Results:
DISCERN (P = 0.001), EQIP (P < 0.001), and JAMA scores (P < 0.001) differed across configurations, whereas GQS did not (P = 0.175). ChatGPT and Copilot had the highest median DISCERN scores (44.00 for both), and DeepSeek had the lowest (35.00); ChatGPT and Copilot each scored significantly higher than DeepSeek. Copilot had the highest median EQIP score (80.00), followed by Perplexity (75.00). Median JAMA scores were 1.00 for ChatGPT, Copilot, and Perplexity and 0.00 for Gemini and DeepSeek. All five models had a median GQS of 4.00. The median clinical factual accuracy score was 5.00 (IQR, 5.00-5.00), and 69 of 75 responses (92.0%) were rated fully accurate. Median completeness was 91.7% (IQR, 87.5%-100.0%). No response contained a potentially harmful recommendation; critical omissions were found in 6 of the 50 applicable responses (12.0%). All configurations had median grade-level scores >6 and median FRES <80. No model met the prespecified sixth-grade readability standard.
Conclusion:
The five LLMs generally produced clinically accurate information about RP. No potentially harmful recommendations were identified. Yet information quality and source transparency differed among models, some responses omitted critical clinical information, and readability was poor across all five. LLMs are currently better suited as professionally reviewed adjuncts to patient education than as substitutes for specialist assessment or individualized clinical decision-making.
