Related Experiment Video
Updated: Jun 23, 2025
09:55
An In Vivo Murine Sciatic Nerve Model of Perineural Invasion
Published on: April 23, 2018
12.1K
Performance of Artificial Intelligence Content Detectors Using Human and Artificial Intelligence-Generated Scientific
Madelyn A Flitcroft1, Salma A Sheriff1, Nathan Wolfrath1
1Department of Surgery, Division of Surgical Oncology, Medical College of Wisconsin, Milwaukee, WI, USA.
Annals of Surgical Oncology
|June 22, 2024
Summary
Publicly available artificial intelligence (AI) content detectors showed varied performance in identifying AI-generated scientific articles. Human-written articles were sometimes misclassified, highlighting the need for continuous evaluation of these AI detection tools.
Area of Science:
- Scientific writing
- Artificial intelligence (AI) content detection
- Academic publishing
Background:
- Limited research exists on the efficacy of AI content detection in scientific writing.
- This study assesses the performance of AI content detectors on human-written and AI-generated scientific articles.
Purpose of the Study:
- To evaluate the accuracy of publicly available AI content detection tools.
- To compare the detection rates for human-written versus AI-generated scientific manuscripts.
Main Methods:
- Analysis of 449 human-written articles from Annals of Surgical Oncology (2022) and 47 AI-generated articles using ChatGPT.
- Evaluation by three AI content detectors, assessing full manuscripts and individual sections.
- Statistical analysis using ANOVA, linear regression, and area under the curve (AUC) for classification performance.
Main Results:
- Human-written articles showed an average AI-generation probability of 9.4%, with significant detector variability.
- Only 0.4% of human-written articles were consistently identified as 0% AI-generated by all detectors.
- AI-generated articles had a higher average AI-generation probability (43.5%), ranging from 12.0% to 99.9%.
Conclusions:
- AI content detectors exhibit performance differences, with a risk of misclassifying human-written content.
- Continuous evaluation and validation are crucial for AI detectors due to the rapid evolution of AI models and detection technologies.
Related Concept Videos
01:16Non-equilibrium in the Cell
4.4K
An important concept in studying metabolism and energy is that of chemical equilibrium. Most chemical reactions are reversible. They can proceed in both directions, releasing energy into their environment in one direction, and absorbing it from the environment in the other direction. The same is true for the chemical reactions involved in cell metabolism, such as the breaking down and building up of proteins into and from individual amino acids, respectively. Reactants within a closed system...
4.4K
01:10Detection of Black Holes
2.2K
Although black holes were theoretically postulated in the 1920s, they remained outside the domain of observational astronomy until the 1970s.
Their closest cousins are neutron stars, which are composed almost entirely of neutrons packed against each other, making them extremely dense. A neutron star has the same mass as the Sun but its diameter is only a few kilometers. Therefore, the escape velocity from their surface is close to the speed of light.
Not until the 1960s, when the first neutron...
Their closest cousins are neutron stars, which are composed almost entirely of neutrons packed against each other, making them extremely dense. A neutron star has the same mass as the Sun but its diameter is only a few kilometers. Therefore, the escape velocity from their surface is close to the speed of light.
Not until the 1960s, when the first neutron...
2.2K
01:00Detection of Gross Error: The Q Test
6.1K
When one or more data points appear far from the rest of the data, there is a need to determine whether they are outliers and whether they should be eliminated from the data set to ensure an accurate representation of the measured value. In many cases, outliers arise from gross errors (or human errors) and do not accurately reflect the underlying phenomenon. In some cases, however, these apparent outliers reflect true phenomenological differences. In these cases, we can use statistical methods...
6.1K
02:16Stereotype Content Model
14.7K
The Stereotype Content Model (SCM) was first proposed by Susan Fiske and her colleagues (Fiske, Cuddy, Glick & Xu, 2002; see also Fiske, 2012 and Fiske, 2017). The SCM specifies that when someone encounters a new group, they will stereotype them based on two metrics: warmth—or that group’s perceived intent, and how likely they are to provide help or inflict harm—and competence—or their ability to carry out that objective. Depending on the warmth-competence...
14.7K
01:05Difference from Background: Limit of Detection
6.3K
The limit of detection (LOD) is the smallest amount of analyte that can be distinguished from the background noise. The LOD value corresponds to the concentration at which the analyte signal is three times larger than the standard deviation of the blank signal. Below this value, the analyte signal cannot be differentiated from the background noise. It is calculated by dividing the calibration slope by 3 times the standard deviation of the blank signals.
The LOD indicates the presence or absence...
The LOD indicates the presence or absence...
6.3K
01:16Deductive Reasoning
55.2K
Deductive reasoning, or deduction, is the type of logic used in hypothesis-based science. In deductive reasoning, the pattern of thinking moves in the opposite direction as compared to inductive reasoning, which means that it uses a general principle or law to predict specific results. From those general principles, a scientist can deduce and predict the specific results that would be valid as long as the general principles are valid.
For example, a researcher can deduce specific predictions...
For example, a researcher can deduce specific predictions...
55.2K