Evaluating Artificial Intelligence-Based Writing Assistance Among Published Orthopaedic Studies: Detection and Trends...

Area of Science:

• Orthopaedic research
• Scientific writing
• Artificial Intelligence

Background:

• The rise of artificial intelligence (AI), especially large language models (LLMs), prompts ethical considerations in scientific writing.
• Interpreting AI detection percentages in orthopaedic literature requires clear statistical thresholds.

Purpose of the Study:

• To quantify AI involvement in published orthopaedic manuscripts.
• To establish a statistical threshold for interpreting AI detection percentages in scientific writing.

Main Methods:

• Analyzed 300 pre-AI-era manuscripts (published in 2000) for AI-generated content.
• Assessed 3,374 post-ChatGPT orthopaedic manuscripts using ZeroGPT.
• Calculated a 95% confidence interval to define a threshold for significant AI involvement (32.875%).

Main Results:

• Pre-AI era manuscripts showed a mean AI detection of 10.84% ± 11.02%.
• 16.7% of post-AI era orthopaedic manuscripts exceeded the 32.875% AI detection threshold.
• Significant AI involvement varied by journal, from 5.6% to 38.3%.

Conclusions:

• AI significantly impacts 16.7% of recently published orthopaedic literature.
• This study provides the first evidence-based threshold for AI detection percentages.
• Emphasizes the need for ethical AI use guidelines and improved detection tools in research.