How appropriately can generative artificial intelligence platforms, including GPT-4, Gemini, Bing, and Wrtn, answer...

This study aims to assess the performance of 4 generative artificial intelligence (AI) platforms-Gemini (formerly Bard), Bing, GPT-4, and Wrtn-in answering questions about colon cancer in the Korean language. Two main research questions guided this study. First, which AI platform provides the most accurate answers? Second, can these AI-generated answers be reliably used to educate patients and their families about colon cancer?

Ten questions selected by the author were posed to the 4 generative AI platforms on February 22, 2024. Two colorectal surgeons in Korea, each with over 20 years of clinical experience, independently evaluated the answers provided by these generative AI platforms.

The generative AI platforms scored an average of 5.5 out of 10 points. Wrtn achieved the highest score at 6 points, followed by GPT-4 and Gemini, each with 5.5, and Bing, scoring 5 points. The weighted κ for inter-rater reliability was 0.597 (P<0.001). The generative AI platforms performed well in explaining the occult blood test for cancer screening, keyhole surgery, and dietary recommendations for cancer prevention. However, they demonstrated significant limitations in answering more complex topics, such as estimating survival rates following surgery, choosing targeted therapy after surgery, and accurately reporting the mortality rate due to colon cancer in Korea.

The findings suggest that using these generative AI platforms as educational resources for patients and their families regarding colon cancer is premature. Further training on colorectal diseases is required before these AI platforms can be considered reliable information sources for the general public in Korea.