Effectiveness of ChatGPT to provide esophageal cancer information: A SERVQUAL-based analysis

ChatGPT and other AI-driven language models are increasingly used in healthcare for disseminating medical information. However, their performance in providing accurate and empathetic responses to patients with specific diseases remains uncertain.

This study aimed to evaluate the effectiveness and reliability of ChatGPT in providing esophageal cancer-related information using the SERVQUAL framework, focusing on five dimensions: Tangibles, Reliability, Responsiveness, Assurance, and Empathy.

Ten representative questions on esophageal cancer were selected through search engine analysis and specialist consultation. ChatGPT generated responses, which were evaluated by 48 gastroenterologists using a 5-point Likert scale aligned with the SERVQUAL model. Statistical analysis was conducted using R 4.4.0 to compare responses between specialist and non-specialist physicians.

ChatGPT performed well in providing structured, logical, and generally informative responses, particularly in the prevention domain. However, limitations were identified in its responsiveness and empathy. Significant differences were observed between specialists and non-specialists in evaluating certain answers, especially regarding reliability and cutting-edge knowledge. ChatGPT often failed to reflect the latest clinical guidelines or regional risk-specific recommendations.

While ChatGPT shows potential in patient education for esophageal cancer, its current outputs lack clinical specificity and up-to-date medical insight. AI tools should be continuously improved with dynamic data integration and specialist supervision to ensure reliability and relevance in real-world healthcare scenarios.