Evaluating Smart Assistant Responses for Accuracy and Misinformation Regarding Human Papillomavirus Vaccination:...

Area of Science:

• Health Informatics
• Public Health
• Artificial Intelligence in Healthcare

Background:

• Smart assistants are increasingly used for health information retrieval, with 46% of Americans using some form and 25% using stand-alone devices.
• The accuracy of health-related information provided by smart assistants requires rigorous evaluation.

Purpose of the Study:

• To evaluate the accuracy, misinformation, and sentiment of smart assistant responses to human papillomavirus (HPV) vaccination queries.
• To compare the performance of major smart assistants in providing HPV vaccine information.

Main Methods:

• Systematic examination of responses to 10 HPV vaccine-related questions across Apple Siri, Google Assistant, Amazon Alexa, and Microsoft Cortana.
• Independent coding of responses by two raters (κ=0.85) to assess accuracy, misinformation, and sentiment.
• Statistical analysis to identify differences in response quality among smart assistants.

Main Results:

• Over half (61%) of the 103 evaluated responses were accurate, with statistically significant differences observed across smart assistants (χ²₂=7.807, P=.02).
• Apple Siri provided the highest proportion of accurate responses (72%), while Microsoft Cortana provided the lowest (54%) and the greatest proportion of misinformation.
• Amazon Alexa demonstrated the most reliable responses, and Cortana's responses had the largest proportion of negative sentiment (12%).

Conclusions:

• Smart assistants offer average-quality sources for HPV vaccination information, with Amazon Alexa being the most reliable.
• Microsoft Cortana exhibited the highest rates of inaccurate responses, misinformation, and negative sentiment.
• Enhanced collaboration between technology companies and public health organizations is crucial for improving the accuracy of health information delivered via smart assistants.