Can AI Tools Reliably and Effectively Detect Plagiarism in Scientific Writing?

Background and aim Artificial intelligence (AI), particularly large language models (LLMs) like ChatGPT (San Francisco, CA: OpenAI) and Bard (now Gemini) (Mountain View, CA: Google), is increasingly used in scientific writing. However, its rapid adoption has raised ethical concerns, especially regarding plagiarism. Current standards of plagiarism detection for scientific writing require a costly and laborious process, done by journal reviewers, sparking questions of whether free LLM tools could streamline the process. Further, AI-generated text can closely resemble genuine scientific writing, raising questions about authenticity and detection. Although various tools exist to identify AI-generated content, their effectiveness remains uncertain. The objective of this study is to evaluate the ability of free online AI tools to identify plagiarism in scientific papers. Methods The following three topics were used for this study: 3D evaluation of the anterior mandible, low-dose cone beam CT, and ameloblastoma case reports. Plagiarized "mashup" papers were created by combining paragraphs from published papers - two papers on 3D evaluation, three on low-dose CBCT, and two on ameloblastoma, each with six paragraphs. These mashups were tested for plagiarism using ChatGPT 3.5 and Bard (five times each) and SmallSEO (London, UK: SmallSEO Tools) (three times). ChatGPT and Bard were then prompted to rewrite the plagiarized mashups, and the rewrites were retested for plagiarism and evaluated for AI detection. Results ChatGPT was unable to identify plagiarism (0/15). Bard detected plagiarism in 8/15 trials, but never identified all plagiarized text. SmallSEO identified 100% of the plagiarism and correctly sourced it, but after AI rewrites, SmallSEO missed 87/90 plagiarized paragraphs. Neither AI-detection tool could definitively detect AI-generated rewrites, with likelihoods never exceeding 70%. ChatGPT and Bard were unable to reliably detect plagiarism. AI-rewritten content was undetectable by plagiarism checkers, and AI-detection tools could not definitively identify AI-generated text. Conclusion ChatGPT, Bard, and SmallSEO are currently unable to identify plagiarism in scientific text. Further, these generative AI tools are capable of rewriting plagiarized text to evade plagiarism detection. Finally, AI-detection tools cannot reliably detect the use of AI in AI-rewritten text.