Avoiding Unintended Consequences of Pediatric Blood Order Set Updates through In Situ Usability Testing

Blood product ordering is a complex process and mistakes can harm patients and lead to poor outcomes. Orders and order sets can be designed to help mitigate errors, but major changes in design can unintentionally cause new errors.

Our objective was as follows: (1) utilize formative in situ usability testing to iteratively improve the design of a redesigned blood product order set prior to go-live, (2) implement changes based on feedback derived from this testing, and (3) compare the error rate, system usability scale (SUS) score, time to task completion, and click counts between the prior order set in use at the time and the revised redesigned order set.

A multidisciplinary project team convened to redesign blood product orders and order sets from scratch based on a review of the literature and benchmarking against four pediatric academic institutions with the goal of addressing prior ordering errors. The newly redesigned blood product order set was iteratively updated via in situ formative usability testing performed with available clinical users using a concurrent think-aloud protocol in real clinical environments. Errors, SUS scores, time to task completion, and click counts were assessed for the revised redesigned order set using summative testing.

Formative usability testing with 20 participants led to seven design changes in the redesigned order set which reduced the error rate at go-live. Summative usability testing showed that even though the usability scores were only slightly improved for the revised redesigned order set, the error rates in blood orders were significantly decreased.

Usability testing can identify design errors early in the process which can be rectified prior to implementation, thus avoiding unintended consequences of changes.