Assessing ChatGPT and Gemini Responses to Common Patient Questions Regarding Augmentation Mammaplasty

Large language models (LLMs) are becoming a common source of medical information for patients.

This study aimed to evaluate and compare the quality and readability of ChatGPT and Google's Gemini in answering frequently asked questions (FAQs) about augmentation mammaplasty (AM).

Ten AM FAQs were submitted to ChatGPT (GPT-4.1 mini) and Gemini (2.5 Flash). Responses were de-identified and independently rated by two board-certified plastic surgeons and one senior resident using the Global Quality Score (GQS). Readability was assessed using the Flesch Reading Ease (FRE) and Flesch-Kincaid Grade Level (FKGL). Paired comparisons used the Wilcoxon signed-rank test for per-question median GQS, inter-rater agreement used Kendall's W, and readability used paired tests as appropriate.

Across 60 individual ratings (3 raters × 10 items × 2 models), per-question median GQS was 5 for 9/10 ChatGPT answers and 10/10 Gemini answers; the paired comparison showed no significant difference (Wilcoxon Z = -1.00; p = 0.317; effect size r = 0.32). Inter-rater agreement was W = 0.24 (ChatGPT, p = 0.091) and W = 0.60 (Gemini, p = 0.002). ChatGPT produced more readable outputs (FRE: 46.53 vs 43.70, p = 0.243; FKGL: 9.71 vs 11.43, p = 0.002), indicating approximately two US grade levels of easier reading.

ChatGPT and Gemini both generated high-quality answers to common AM FAQs, with no difference in quality based on GQS. ChatGPT's responses were significantly easier to read according to FKGL. LLMs may support patient education when implemented with clinician oversight to mitigate limitations and prevent misinformation.

This journal requires that authors assign a level of evidence to each article. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors www.springer.com/00266 .