Artificial Intelligence Chatbot Accuracy for Patient Education in HPV Infection

Chatbots are artificial intelligence (AI) tools that use large language models to answer questions and simulate human-like conversations. Chatbots are commonly used by patients for self-education regarding medical issues. The accuracy of information obtained through patient self-education searches using Chatbots for Human Papillomavirus (HPV) infection and recurrent respiratory papillomatosis (RRP) is unknown. This work is intended to analyze aspects of chatbot answers to common patient questions regarding laryngeal HPV infection and RRP.

Eight questions were asked by the authors to mimic real-world patient queries to three different Chatbot models (Claude, ChatGPT, and Gemini). Two fellowship-trained laryngologists blindly rated eight answers for each model using a 5-point Likert scale regarding accuracy, comprehensiveness, and similarity to a response the rating physician would themselves offer. Descriptive and statistical analyses were performed.

Gemini had the highest overall mean score (4.7, SD 0.5), and Claude had the lowest (4, SD 0.7). Gemini had the highest individual scores for each aspect evaluated. Between the aspects analyzed across all platforms, accuracy was scored highest (4.4, SD 0.6), but the difference was not robust. The one-way ANOVA test showed an overall significant difference between the three platforms (P < 0.0001). A paired comparison between platforms was performed using a post hoc Tukey test, and the results were only significant when Gemini was compared individually with ChatGPT and Claude (P < 0.0001).

Gemini had the highest overall and individual rated scores for answer accuracy, comprehensiveness, and similarity to what physician raters would offer their patients. It was the only chatbot platform that demonstrated a robust difference when compared to the others regarding answers to common patient questions on laryngeal HPV infection/RRP.