Related Experiment Videos
Comparing Human and AI-Generated Editorials in Orthopaedic Oncology: A Blinded Evaluation of Quality and Authorship
Jessa Fogel1, Ahlam Ashkar, Philip Khoury
1From the School of Medicine (Fogel, Ashkar, Khoury, Ng, Ghert), University of Maryland, Baltimore, MD, the Université Catholique de Louvain (Schubert), Ottignies-Louvain-La-Neuve, Belgium, the Division of Orthopaedic Surgery (Bozzo), McGill University, Montreal, QC, Canada, the UCLA Department of Orthopaedic Surgery (Christ), Los Angeles, CA, the Department of Orthopaedics (Dr. Choong), University of Melbourne, Melbourne, Australia, the Hospital Universitario Austral (HUA) (Serra), Universidad Austral, Pilar, Argentina, the Division of Orthopaedic Oncology (Pretell), Miami Cancer Institute, Baptist Health South Florida, Plantation, FL, the Ganga Medical Center and Hospitals Pvt. Ltd (Rajasekaran), Coimbatore, India, the Department of Orthopaedics (Kumar), India Institute of Medical Sciences, New Delhi, India, the Department of Orthopaedics (Levin), Johns Hopkins University, Baltimore, MD, the OHSU Department of Orthopaedics (Gundle), Portland, OR, the Parkview Regional Medical Center (Johnson), Fort Wayne, IN, the Hospital de Clínicas de Porto Alegre (Becker), Porto Alegre, Brazil, the Department of Sarcoma (Lazarides), Moffitt Cancer Center, Tampa, FL, the Orthopaedic Department (El Ghoneimy), Children's Cancer Hospital Egypt, Cairo, Egypt, the Acute and Reparative Medicine (Tsagkozis), Karolinska University Hospital, Stockholm, Sweden, the Virginia Cancer Specialists (Wodajo), University of Virginia School of Medicine, Inova Campus, Charlottesville, VA, the Unidad de Reconstrucción Osteoarticular y Tumores (Vélez), Servicio de Cirugía Ortopédica y Traumatología, Hospital Universitari Vall d'Hebron, Barcelona, Spain, the Institute for Interdisciplinary Sarcoma Treatment and Research (Andreou), Department of Orthopaedic Oncology and Sarcoma Surgery, University Hospital Essen, West German Cancer Center, Essen, Germany, the Department of Orthopaedics and Trauma (Leithner), Medical University of Graz, Graz, Austria, and the Allegheny Health Network Orthopaedic Institute (Ercolano), Pittsburgh, PA.
Background:
As the use of artificial intelligence (AI) and large language models (LLMs) is increasingly adopted into scientific writing, it is important to understand AI's ability to produce clear and accurate content that is on par with human-authored content in the field of orthopaedics, including orthopaedic oncology. The aim of this study was to compare a series of editorials written by orthopaedic oncologists with those written by a single LLM (ChatGPT 4.0) using a variety of quality metrics.
Methods:
Volunteer orthopaedic oncologists submitted a 3- to 4-paragraph persuasive editorial on a topic of their choice in the field of musculoskeletal oncology. ChatGPT 4.0 was then prompted to write a corresponding editorial for each topic. Each editorial was evaluated by two blinded peer reviewers and graded using a 25-point scale on the following quality metrics: content, clarity, grammar, persuasiveness, and creativity. The evaluators were also asked to indicate whether they believed the editorials were written by humans or by AI.
Results:
A total of 20 editorials were submitted by human authors and matched with 20 prompted AI editorials. No notable difference in average total quality score for human versus AI submissions was observed. AI-generated articles scored markedly higher in grammar, but there were no notable differences in any other quality metric. Reviewers correctly identified author type 59% of the time.
Discussion:
LLMs such as ChatGPT can generate editorial content in orthopaedic oncology that matches human-written quality, suggesting a potential supportive role for AI in scientific communication, with implications for authorship standards, editorial practices, and peer review.
Related Concept Videos
Cancer
Bone Remodeling
The Retina
Bone Structure
Vision
Altruism