Use of ChatGPT for patient education involving HPV-associated oropharyngeal cancer

This study aims to investigate the ability of ChatGPT to generate reliably accurate responses to patient-based queries specifically regarding oropharyngeal squamous cell carcinoma (OPSCC) of the head and neck.

Retrospective review of published abstracts.

Publicly available generative artificial intelligence.

ChatGPT 3.5 (May 2024) was queried with a set of 30 questions pertaining to HPV-associated oropharyngeal cancer that the average patient may ask. This set of questions was queried a total of four times preceded by a different prompt. The answer prompts for each question set were reviewed and graded on a four-part Likert scale. A Flesch-Kincaid reading level was also calculated for each prompt.

For all answer prompts (n = 120), 6.6 % were graded as mostly inaccurate, 7.5 % were graded as minorly inaccurate, 41.7 % were graded as accurate, and 44.2 % were graded as accurate and helpful. The average Flesch-Kincaid reading grade level was lowest for the responses without any prompt (11.77). Understandably, the highest grade levels were found in the physician-friend prompt (12.97). Of the 30 references, 25 (83.3 %) were found to be authentic published studies. Of the 25 authentic references, the answers accurately cited information found within the original source for 14 of the references (56 %).

ChatGPT was able to produce relatively accurate responses to example patient questions, but there was a high rate of false references. In addition, the reading level of the answer prompts was well above the Centers for Disease Control and Prevention (CDC) recommendations for the average patient.