Large Language Model-based Chatbot as a Source of Advice on First Aid in Heart Attack.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Emergency Medicine

Background:

• Large language models (LLMs) show promise for generating human-like responses.
• Hypothetically, LLM-powered chatbots could provide real-time first aid advice for cardiovascular emergencies.
• The reliability of such AI for critical health information requires thorough evaluation.

Purpose of the Study:

• To assess the quality of chatbot responses regarding first aid for heart attack scenarios.
• To compare chatbot-generated advice against established international first aid guidelines.
• To identify omissions, inaccuracies, and misleading information in AI-driven health advice.

Main Methods:

• Simulated user inquiries to the new Bing chatbot using a "heart attack what to do" prompt from the Gambia, India, and the USA.
• Evaluation of 20 chatbot responses per country against the International First Aid, Resuscitation, and Education Guidelines 2020 using a checklist.
• Analysis of response congruence, omissions of life-saving instructions, and inclusion of superfluous or inconsistent directives.

Main Results:

• Chatbot responses provided some first aid guidance but commonly omitted critical steps like encouraging rest or initiating cardiopulmonary resuscitation (CPR).
• Mean response congruence with guidelines ranged from 7.3% (India) to 16.8% (USA).
• Significant portions of responses contained superfluous or inconsistent directives: 45% in India, and 25% in the Gambia and USA.

Conclusions:

• Current chatbot advice on heart attack first aid is unreliable due to omissions, inaccuracies, and misleading instructions.
• Chatbots cannot be recommended as credible sources for emergency first aid information at this time.
• Further research and organizational efforts are crucial to ensure the trustworthiness and safety of AI-mediated health counseling.