Artificial intelligence chatbots in response to patient's common inquiries about chordoma: A cross-sectional study

Artificial intelligence (AI) chatbots has been well studied in many common diseases. However, little was reported for chordoma, which is a rare disease with high rates of recurrence, disability, and mortality.

This study aims to assess the performance of oncologists and state-of-the-art AI chatbots in response to frequently-asked questions (FAQs) of chordoma from real world. The response performance was mainly evaluated by overall quality, empathy, and readability.

Sixty chordoma-related FAQs, collected from social media, were addressed by various chatbots and oncologists, and the best-performing chatbot-generated text was further edited and assessed again. Rated scores were ordered for quality and empathy in a blind way. The readability was measured objectively by calculating Flesch-Kincaid Grade Level (FKGL), Automated Readability Index (ARI), and Gunning-Fog Index (GFI).

AI chatbots were universally superior to oncologists in response quality (3.86 ± 0.14 vs. 3.12 ± 0.25, p < 0.001) and empathy (3.28 ± 0.41 vs. 2.95 ± 0.48, p < 0.001). DeepSeek-R1 achieved highest rated score in response quality (4.20 ± 0.22), while Claude 3.5 Sonnet was considered as the best chatbots by comprehensive assessments. The chatbot drafted responses were easier to understand from patient's perspective (p < 0.001). Improved response quality (4.09 ± 0.12, p < 0.001), empathy (4.00 ± 0.39, p < 0.001), and readability (FKGL: 11.30 ± 2.42, p < 0.001) were obtained after editing the Claude-3.5-generated responses by oncologists.

AI chatbots reached favorable quality and empathetic performance in response to chordoma-related FAQs, and generated equivalent readability compared to oncologists. With generative chatbot's assistance, oncologists may response more comprehensively and efficiently in addressing chordoma patient's common inquiries.