Can ChatGPT Recognize Its Own Writing in Scientific Abstracts?

Area of Science:

• Artificial Intelligence in Scientific Writing
• Academic Publishing Integrity

Background:

• The proliferation of generative AI presents challenges in distinguishing AI-generated from human-authored scientific content.
• Assessing the capability of advanced AI models, like ChatGPT-4.0, to self-identify AI-generated text is crucial for maintaining academic standards.

Purpose of the Study:

• To evaluate ChatGPT-4.0's accuracy in distinguishing between human-written and AI-generated scientific abstracts.
• To determine if ChatGPT-4.0 can reliably recognize its own output in a scientific context.

Main Methods:

• 100 original research abstracts from 2000 were selected from high-impact internal medicine journals.
• ChatGPT-4.0 generated new abstracts from the full text of these articles.
• ChatGPT-4.0 evaluated both original and AI-generated abstracts twice, rating authorship likelihood on a 0-10 scale.

Main Results:

• High misclassification rates were observed in both evaluation rounds (49% and 47.5%).
• No abstracts were assigned a neutral score (5), indicating a lack of uncertainty.
• Statistical analysis revealed substantial overlap in score distributions and poor agreement (Cohen's kappa = 0.33, weighted kappa = 0.24).

Conclusions:

• ChatGPT-4.0 demonstrates an inability to reliably distinguish between human and AI-generated scientific abstracts.
• The findings underscore the necessity for developing advanced external tools to verify authorship and ensure transparency in academic publishing.