Accurately detecting AI text when ChatGPT is told to write like a chemist

Area of Science:

• Scientific publishing
• Artificial intelligence
• Text generation

Background:

• Large language models (LLMs) like ChatGPT generate human-like text rapidly.
• Many scientific journals reject AI as authors, necessitating AI text detection.
• Distinguishing AI-generated text from human writing is an immediate need.

Purpose of the Study:

• To test the accuracy of a newly developed AI text detector for scientific journals.
• To evaluate the detector's performance on diverse text types, including challenging AI-generated content.
• To assess the detector's effectiveness in identifying AI text in chemistry journals.

Main Methods:

• Development of an AI text detector specifically for scientific journals.
• Testing the detector on human-written text from various chemistry journals.
• Evaluating the detector on AI-generated text from advanced models (e.g., GPT-4).
• Assessing performance on AI text generated with obfuscation prompts.

Main Results:

• The AI text detector achieved high accuracy in distinguishing between human and AI-generated text.
• The detector successfully identified AI-generated text across all tested scenarios, including challenging cases.
• ChatGPT-generated text within chemistry journals was readily detected.

Conclusions:

• The developed AI text detector is effective in identifying AI-generated scientific text.
• This capability is fundamental for understanding AI's future impact on scientific publishing.
• Accurate AI text detection is essential for maintaining academic integrity.