User-Centered Evaluation of a Visual Annotation Tool for Rapid Assessment of Pediatric Weight Entry Errors

Weight entry errors can cause significant patient harm in pediatrics due to pervasive weight-based dosing practices. While computerized algorithms can assist in error detection, they have not achieved high sensitivity and specificity to be further developed as a clinical decision support tool. To train an advanced algorithm, expert-annotated weight errors are essential but difficult to collect. In this study, we developed a visual annotation tool to gather large amounts of expertly annotated pediatric weight charts and conducted a formal user-centered evaluation. Key features of the tool included configurable grid sizes and annotation styles. The user feedback was collected through a structured survey and user clicks on the interface. The results show that the visual annotation tool has high usability (average SUS=86.4). Different combinations of the key features, however, did not significantly improve the annotation efficiency and duration. We have used this tool to collect expert annotations for algorithm development and benchmarking.