Related Experiment Video
Updated: Jun 23, 2025

An In Vivo Murine Sciatic Nerve Model of Perineural Invasion
Published on: April 23, 2018
Performance of Artificial Intelligence Content Detectors Using Human and Artificial Intelligence-Generated Scientific
Madelyn A Flitcroft1, Salma A Sheriff1, Nathan Wolfrath1
1Department of Surgery, Division of Surgical Oncology, Medical College of Wisconsin, Milwaukee, WI, USA.
Background:
Few studies have examined the performance of artificial intelligence (AI) content detection in scientific writing. This study evaluates the performance of publicly available AI content detectors when applied to both human-written and AI-generated scientific articles.
Methods:
Articles published in Annals of Surgical Oncology (ASO) during the year 2022, as well as AI-generated articles using OpenAI's ChatGPT, were analyzed by three AI content detectors to assess the probability of AI-generated content. Full manuscripts and their individual sections were evaluated. Group comparisons and trend analyses were conducted by using ANOVA and linear regression. Classification performance was determined using area under the curve (AUC).
Results:
A total of 449 original articles met inclusion criteria and were evaluated to determine the likelihood of being generated by AI. Each detector also evaluated 47 AI-generated articles by using titles from ASO articles. Human-written articles had an average probability of being AI-generated of 9.4% with significant differences between the detectors. Only two (0.4%) human-written manuscripts were detected as having a 0% probability of being AI-generated by all three detectors. Completely AI-generated articles were evaluated to have a higher average probability of being AI-generated (43.5%) with a range from 12.0 to 99.9%.
Conclusions:
This study demonstrates differences in the performance of various AI content detectors with the potential to label human-written articles as AI-generated. Any effort toward implementing AI detectors must include a strategy for continuous evaluation and validation as AI models and detectors rapidly evolve.
Related Concept Videos
Non-equilibrium in the Cell
Detection of Black Holes
Their closest cousins are neutron stars, which are composed almost entirely of neutrons packed against each other, making them extremely dense. A neutron star has the same mass as the Sun but its diameter is only a few kilometers. Therefore, the escape velocity from their surface is close to the speed of light.
Not until the 1960s, when the first neutron...
Detection of Gross Error: The Q Test
Stereotype Content Model
Difference from Background: Limit of Detection
The LOD indicates the presence or absence...
Deductive Reasoning
For example, a researcher can deduce specific predictions...

