Beyond text generation: a comprehensive evaluation of ChatGPT-5.5 for scientific narrative review writing on sleeve...

Large language models are increasingly being used in academic and scientific writing, but their reliability in medical literature generation remains uncertain. In particular, concerns persist regarding factual accuracy, reference validity, originality, and the overall academic quality of AI-assisted manuscripts. This study aimed to systematically evaluate the capacity of ChatGPT-5.5 to generate a scientific narrative review on sleeve gastrectomy using structured prompting and objective assessment criteria. ChatGPT-5.5 was instructed to generate a narrative review on long-term metabolic improvement and weight loss outcomes after sleeve gastrectomy using a structured prompt framework. The generated manuscript was independently evaluated by two general surgeons with expertise in bariatric and metabolic surgery. Assessment domains included scientific accuracy, reference validity, plagiarism screening, narrative review quality using the Scale for the Assessment of Narrative Review Articles (SANRA), and academic quality using a structured peer-review rubric. The model generated a structured nine-section review outline and a complete narrative review manuscript. Overall, 85 statements were identified and evaluated. Of the 40 cited statements, 38 were factually correct and 2 were factually incorrect. The remaining 45 uncited statements were also considered factually correct, although 16 were classified as requiring supporting references. Reference verification revealed that 16 of 33 references (48.48%) were fully verifiable, whereas 17 references (51.52%) were inaccurate or unverifiable, including 3 entirely fabricated citations. iThenticate analysis showed an overall similarity score of 14%, with no substantial evidence of plagiarism on manual review. SANRA scores were 7 and 8 out of 12, while structured peer-review rubric scores were 37 and 36 out of 50, indicating overall good manuscript quality. Inter-rater agreement was excellent for both SANRA and rubric scoring. In this structured evaluation, ChatGPT-5.5 generated a narrative review on sleeve gastrectomy that was generally well structured, factually accurate, original, and of acceptable academic quality. These findings suggest that large language models may have a supportive role in surgical academic writing, particularly in organizing review content and generating coherent scientific text. However, important limitations were identified in reference validity, including bibliographic inaccuracies and fabricated citations. These findings suggest that large language models may support scientific writing when guided by structured prompts, but their outputs require expert oversight, reference verification, and critical validation before use in academic publishing.