Human versus artificial intelligence-generated arthroplasty literature: A single-blinded analysis of perceived
Kyle W Lawrence1, Akram A Habibi1, Spencer A Ward1
1Department of Orthopedic Surgery, NYU Langone Health, New York, New York, USA.
Background:
Large language models (LLM) have unknown implications for medical research. This study assessed whether LLM-generated abstracts are distinguishable from human-written abstracts and to compare their perceived quality.
Methods:
The LLM ChatGPT was used to generate 20 arthroplasty abstracts (AI-generated) based on full-text manuscripts, which were compared to originally published abstracts (human-written). Six blinded orthopaedic surgeons rated abstracts on overall quality, communication, and confidence in the authorship source. Authorship-confidence scores were compared to a test value representing complete inability to discern authorship.
Results:
Modestly increased confidence in human authorship was observed for human-written abstracts compared with AI-generated abstracts (p = 0.028), though AI-generated abstract authorship-confidence scores were statistically consistent with inability to discern authorship (p = 0.999). Overall abstract quality was higher for human-written abstracts (p = 0.019).
Conclusions:
AI-generated abstracts' absolute authorship-confidence ratings demonstrated difficulty in discerning authorship but did not achieve the perceived quality of human-written abstracts. Caution is warranted in implementing LLMs into scientific writing.
Related Concept Videos
Reason and Intuition
Stereotype Content Model
Non-equilibrium in the Cell
Social Foundations of Self IV: Self in Digital Communication
Automatic Processing and Automatic Social Behavior
Understanding Deception


