Artificial Intelligence-Generated Editorials in Radiology: Can Expert Editors Detect Them?

Area of Science:

• Artificial Intelligence in Scientific Writing
• Radiology Editorial Content Generation
• Natural Language Processing in Academia

Background:

• Artificial intelligence (AI) models like GPT-4 can produce human-like text.
• The capability of AI to write scientific content requires evaluation for its real-world applicability.

Purpose of the Study:

• To assess GPT-4's ability to generate radiology editorials.
• To compare AI-generated editorials with human-written ones.
• To determine the detectability and publishability of AI-generated scientific content.

Main Methods:

• GPT-4 generated 16 editorials based on human-written summaries.
• Six experienced editors evaluated articles using Likert scales and authorship determination (unpaired and paired settings).
• Articles were analyzed using AI detection and plagiarism software.

Main Results:

• AI-written articles had a median AI probability score of 58%, compared to 2.0% for human-written articles.
• Authorship classification accuracy reached 70% in the paired setting.
• Editors preferred publishing human-written articles (82%) and rated them higher when perceived as human-authored.

Conclusions:

• GPT-4 produces high-quality, plagiarism-free editorials that are difficult to detect with current tools.
• AI-generated content may be published but faces editor bias towards human-written work.
• Detection tools have limited effectiveness in identifying AI-generated scientific text.