Related Experiment Video
Updated: May 6, 2026
A Postoperative Evaluation Guideline for Computer-Assisted Reconstruction of the Mandible
Published on: January 28, 2020
Accuracy and Completeness of Bard and Chat-GPT 4 Responses for Questions Derived from the International Consensus
Yavar Abgin1, Kayla Umemoto1, Andrew Goulian1
1California Northstate University College of Medicine, Elk Grove, California, United States.
Abstract:
Artificial intelligence large language models (LLMs), such as Chat Generative Pre-Trained Transformer 4 (Chat-GPT) by OpenAI and Bard by Google, emerged in 2022 as tools for answering questions, providing information, and offering suggestions to the layperson. These LLMs impact how information is disseminated and it is essential to compare their answers to experts in the corresponding field. The International Consensus Statement on Endoscopic Skull-Base Surgery 2019 (ICAR:SB) is a multidisciplinary international collaboration that critically evaluated and graded the current literature.
Objectives:
Evaluate the accuracy and completeness of Chat-GPT and Bard responses to questions derived from the ICAR:SB policy statements.
Design:
Thirty-four questions were created based on ICAR:SB policy statements and input into Chat-GPT and Bard. Two rhinologists and two neurosurgeons graded the accuracy and completeness of LLM responses, using a 5-point Likert scale. The Wilcoxon rank-sum and Kruskal-Wallis tests were used for analysis.
Setting:
Online.
Participants:
None.
Outcomes:
Compare the mean accuracy and completeness scores between (1) responses generated by Chat-GPT versus Bard and (2) rhinologists versus neurosurgeons.
Results:
Using the Wilcoxon rank-sum test, there were statistically significant differences in (1) accuracy ( p < 0.001) and completeness ( p < 0.001) of Chat-GPT compared with Bard; and (2) accuracy ( p < 0.001) and completeness ( p < 0.001) ratings between rhinologists and neurosurgeons.
Conclusion:
Chat-GPT responses are overall more accurate and complete compared with Bard, but both are very accurate and complete. Overall, rhinologists graded lower than neurosurgeons. Further research is needed to better understand the full potential of LLMs.
