Evaluating the agreement of users with usability problems identified by heuristic evaluation

Area of Science:

• Health Informatics
• Human-Computer Interaction
• Usability Engineering

Background:

• Heuristic evaluation (HE) is a method for identifying usability problems in health information systems.
• Evidence is lacking on how well HE-identified problems align with actual end-user experiences in clinical settings.

Purpose of the Study:

• To assess the agreement between problems identified through heuristic evaluation and those experienced by real end-users of a health information system.

Main Methods:

• Heuristic evaluation was conducted on an emergency information system in Iranian educational hospitals.
• A validated electronic questionnaire (Cronbach's alpha = 0.95) was developed based on HE findings to survey end-users.

Main Results:

• Out of 127 identified problems, "consistency and standards" accounted for 18.9%.
• The overall user agreement coefficient with heuristic evaluation results was 0.5 (medium agreement).
• Highest user agreement (78%) was observed for problems related to "help user recognize, diagnose and recover from errors."

Conclusions:

• The agreement between heuristic evaluation and end-user experience is moderate, suggesting potential discrepancies in problem perception.
• Heuristic evaluation is effective for identifying specific issues like error handling but may overlook critical problems or flag minor ones, especially in areas like design consistency.