Artificial intelligence as author: Can scientific reviewers recognize GPT-4o-generated manuscripts?

Chat Generative Pre-Trained Transformer (ChatGPT) is a natural language processing model. It can be argued that ChatGPT has recently begun to assume the role of a technological assistant capable of supporting academics in the process of scientific writing. ChatGPT may contribute to the spread of incorrect or incomplete information within academic literature, leading to conceptual confusion and potential academic misconduct. The aim of this study is to determine whether a scientific article entirely generated by an AI application such as ChatGPT can be detected by an academic journal editor or peer reviewer.

This study was conducted between November 1, 2024, and December 1, 2024. GPT-4o, was utilized in this study. ChatGPT was instructed to write a scientific article focused on predicting mortality and return of spontaneous circulation (ROSC) in OHCA cases. The manuscript written by ChatGPT-4o was sent to 14 different reviewers who had previously served as reviewers or editors. The reviewers were asked to evaluate the manuscript as if they were an SCI-E journal editor or peer reviewer. The reviewers were informed that the article had been written by ChatGPT and were asked whether they had identified this during their review.

Among the reviewers, 42.9 % (n = 6) decided to reject the manuscript at the editorial stage, whereas another 42.9 % (n = 6) opted to forward it to a peer reviewer. During the peer review stage, 42.9 % (n = 6) of the reviewers recommended rejection, while 28.6 % (n = 4) suggested major revisions. 78.6 % (n = 11) of the reviewers did not realize that the manuscript had been generated by an artificial intelligence model.

The findings of our study highlight the necessity for journal editors and peer reviewers to be well-informed about ChatGPT and to develop systems capable of identifying whether a manuscript has been written by a human or generated by artificial intelligence.