Chat GPT vs an experienced ophthalmologist: evaluating chatbot writing performance in ophthalmology

Area of Science:

• Ophthalmology
• Artificial Intelligence in Medicine
• Scientific Writing

Background:

• The rapid advancement of artificial intelligence (AI) necessitates evaluating its capabilities in specialized scientific fields.
• Assessing AI's role in generating scientific content is crucial for understanding its potential impact on research and publication.

Purpose of the Study:

• To evaluate the proficiency of ChatGPT in drafting scientific introductions for ophthalmology research papers.
• To compare the quality of ChatGPT-generated introductions against those written by experienced ophthalmologists.

Main Methods:

• ChatGPT 4 was prompted to generate introductions for selected ophthalmology papers.
• Ten experienced ophthalmology specialists evaluated introductions written by ChatGPT and human authors, without knowing the source.
• Evaluations focused on metrics including language, data arrangement, factual accuracy, originality, and data currency.

Main Results:

• Ophthalmology specialists correctly identified the source of introductions in only 57.7% of cases.
• No significant differences were found between ChatGPT and human authors across assessed quality metrics.
• Misclassification rates varied by subspecialty, with Oculoplastics having the highest (66.7%) and Retina the lowest (11.1%).

Conclusions:

• ChatGPT demonstrates significant capability in generating scientific paper introductions in ophthalmology.
• AI-generated introductions are statistically indistinguishable from expert-written ones in key quality aspects.
• Further research should explore AI's utility in other manuscript sections and address ethical implications.