Are three methods better than one? A comparative assessment of usability evaluation methods in an EHR

Area of Science:

• Health Informatics
• Human-Computer Interaction
• Software Engineering

Background:

• Usability is critical for the effective adoption and use of Electronic Health Records (EHRs).
• Identifying usability problems in EHR systems is essential for improving user experience and patient care.
• Various end-user feedback methods exist, each with potential strengths and limitations.

Purpose of the Study:

• To comparatively evaluate the effectiveness of user testing, semi-structured interviews, and surveys for detecting EHR usability problems.
• To determine the complementarity of these methods in identifying a comprehensive set of usability issues.

Main Methods:

• Data collected from faculty, residents, and dental students at two urban dental schools.
• Three methods employed: user testing (N=32), semi-structured interviews (N=36), and surveys (N=35).
• Focus on assessing the usability of a dental EHR for treatment plan development.

Main Results:

• A total of 187 usability violations were identified across all methods.
• User testing detected the majority of violations (54%), followed by interviews (28%) and surveys (18%), with limited overlap.
• User testing covered the widest range of usability themes (83%), compared to interviews (63%) and surveys (29%).

Conclusions:

• A multi-method approach provides a more comprehensive evaluation of EHR usability challenges.
• User testing, interviews, and surveys are complementary, offering unique insights for software enhancement.
• Interview and survey methods alone are insufficient but valuable when combined with user testing for a thorough EHR evaluation.