A comparison of ChatGPT-generated articles with human-written articles

Area of Science:

• Artificial Intelligence in Medicine
• Scientific Publishing

Background:

• ChatGPT, an AI language tool, generates human-like text.
• Concerns exist regarding the accuracy of AI-generated academic content.

Purpose of the Study:

• To compare the accuracy and quality of ChatGPT-generated academic articles against human-authored ones.
• To assess the reliability of AI-generated scientific literature.

Main Methods:

• Evaluated ChatGPT-generated radiology articles against published/under-review works.
• Two fellowship-trained musculoskeletal radiologists independently graded articles on a 1-5 accuracy scale.

Main Results:

• Four out of five ChatGPT articles were significantly inaccurate with fictitious references.
• One article was well-written but contained entirely fabricated references.
• AI-generated articles may appear authentic to non-expert readers.

Conclusions:

• ChatGPT can produce coherent research articles that mimic authentic publications.
• All assessed ChatGPT articles contained factual inaccuracies and fake references.
• The reliability of AI-generated scientific content requires critical evaluation.