Confidence without consensus? Human versus AI abstracts in urology

This study asked whether urologists can tell human-written abstracts apart from artificial intelligence (AI)-written abstracts when both describe the same study, and whether the answer depends on which AI model did the writing.

Fifteen published urology studies were selected. For each study, the original human-written abstract was paired with three AI-written abstracts covering the same material, produced by Claude Sonnet 4.6, GPT-5.5, and Gemini 3 Flash using identical instructions and each model's ordinary consumer settings. Because all four versions described the same study, the underlying data were held constant, and the versions differed in how that material was written and presented. The resulting 60 abstracts were stripped of identifying information, and eight urologists read all of them without knowing which was which (480 judgments in total). For each abstract, reviewers stated whether they believed it was written by a human or by AI, rated how confident they were in that judgment, and rated the abstract's quality and credibility on 5-point scales.

Reviewers correctly identified AI-written abstracts 49.3% of the time, approximately 50%, compared with 62.4% for human-written abstracts (p = .011). Results depended heavily on which model wrote the text (p = .003). Abstracts written by Claude were recognized as AI only 40.0% of the time, significantly less often than human abstracts were correctly identified (p = .003) and less often than GPT abstracts (56.8%, p = .038); Gemini fell between the two at 50.9%. Quality ratings differed only for GPT, which scored lower than human abstracts (p = .046; omnibus p = .030); credibility ratings did not differ between sources at all (p = .577). Reviewers who felt confident were no more likely to be correct (p = .749), and reviewers seldom agreed with one another about which abstracts were AI-written (Fleiss' κ = -0.009).

Urologists could not reliably recognize AI-written urology abstracts, and the difficulty came mainly from one model rather than from AI writing in general. How good an abstract appeared had little to do with whether its authorship was correctly identified, and neither reviewer confidence nor agreement between reviewers gave any warning. Reading alone is therefore not a dependable way to detect undisclosed AI authorship.