Assessing the reliability of AI-generated responses to patient-oriented lung cancer questions: a multi-platform study

Large language models (LLMs) are increasingly used by patients seeking lung cancer-related medical information. Recent studies have increasingly compared multiple LLMs for patient-facing medical information; however, differences in clinically relevant performance dimensions and consistency across repeated responses remain important areas for evaluation.

Seventy-nine lung cancer-related patient questions were queried across three freely accessible LLM platforms (ChatGPT, Claude, and Google Gemini) between December 2025 and January 2026. Responses were independently evaluated by two specialist thoracic surgeons for accuracy, completeness, clinical relevance, and readability, with disagreements resolved by consensus. Reproducibility was assessed through five repeated independent queries using a clinical-equivalence framework, and automated readability was assessed using the Flesch-Kincaid Grade Level. Ordinal outcomes were analyzed using cumulative link mixed models accounting for repeated evaluation of questions, while reproducibility was analyzed using generalized estimating equations.

All platforms demonstrated high accuracy, with no significant overall platform difference (p = 0.150). Completeness differed significantly across platforms (p < 0.001), with higher ratings for ChatGPT than Claude and Gemini and for Claude than Gemini. Clinical relevance showed no overall platform effect (p = 0.568), although a platform-by-domain interaction was observed (p = 0.046). Expert-rated readability differed significantly (p < 0.001), with Gemini receiving higher ratings than ChatGPT and Claude. All platforms produced content requiring approximately 10th-12th grade reading levels. Reproducibility was high overall (88.6-96.2%), with no statistically significant overall platform effect (p = 0.072).

Contemporary LLMs demonstrated high accuracy and clinical relevance, but differed in completeness and readability. Reproducibility was high across platforms, while all models exceeded recommended patient reading levels. Further patient-based validation is needed before routine clinical integration.

Not applicable.