A Comprehensive Comparison and Evaluation of AI-Powered Healthcare Mobile Applications' Usability

Area of Science:

• Digital Health
• Human-Computer Interaction
• Artificial Intelligence in Medicine

Background:

• Proliferation of AI symptom checker apps in mHealth.
• Limited examination of their real-world usability and transparency.

Purpose of the Study:

• Triangulated evaluation of three AI-powered mHealth apps: ADA, Mediktor, and WebMD.
• Assessing usability and transparency through expert and lay user perspectives.

Main Methods:

• Usability experts used an AI-specific heuristic checklist.
• Lay users (n=30) completed health scenarios, with task success, errors, and completion time recorded.
• System Usability Scale (SUS) scores compared across apps using ANOVA and t-tests.

Main Results:

• Statistically significant usability differences found across apps (p < 0.001).
• ADA achieved higher SUS scores than Mediktor and WebMD; Mediktor outperformed WebMD.
• Common issues: vague AI outputs, poor error feedback, inconsistent navigation, and lack of explainability.

Conclusions:

• Even top-rated AI mHealth apps have critical explainability and error-handling deficits.
• Explainable AI (XAI) features like confidence indicators and transparent justifications are crucial.
• Enhancing XAI can improve user trust, safety, and adoption in healthcare.