Related Experiment Video
Updated: Jan 14, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Evaluating chat generative pretrained transformer in answering questions on endoscopic mucosal resection and
Shi-Song Wang1,2, Hui Gao1, Peng-Yao Lin2
1Department of Gastroenterology, The First Affiliated Hospital of Ningbo University, Ningbo 315010, Zhejiang Province, China.
Background:
With the rising use of endoscopic submucosal dissection (ESD) and endoscopic mucosal resection (EMR), patients are increasingly questioning various aspects of these endoscopic procedures. At the same time, conversational artificial intelligence (AI) tools like chat generative pretrained transformer (ChatGPT) are rapidly emerging as sources of medical information.
Aim:
To evaluate ChatGPT's reliability and usefulness regarding ESD and EMR for patients and healthcare professionals.
Methods:
In this study, 30 specific questions related to ESD and EMR were identified. Then, these questions were repeatedly entered into ChatGPT, with two independent answers generated for each question. A Likert scale was used to rate the accuracy, completeness, and comprehensibility of the responses. Meanwhile, a binary category (high/Low) was used to evaluate each aspect of the two responses generated by ChatGPT and the response retrieved from Google.
Results:
By analyzing the average scores of the three raters, our findings indicated that the responses generated by ChatGPT received high ratings for accuracy (mean score of 5.14 out of 6), completeness (mean score of 2.34 out of 3), and comprehensibility (mean score of 2.96 out of 3). Kendall's coefficients of concordance indicated good agreement among raters (all P < 0.05). For the responses generated by Google, more than half were classified by experts as having low accuracy and low completeness.
Conclusion:
ChatGPT provided accurate and reliable answers in response to questions about ESD and EMR. Future studies should address ChatGPT's current limitations by incorporating more detailed and up-to-date medical information. This could establish AI chatbots as significant resource for both patients and health care professionals.
