Mixed Reality and Desktop Hand Hygiene Training With Deep Learning-Based Step Recognition and Real-Time Decision...

Hand hygiene (HH) is essential for preventing healthcare-associated infections, yet conventional monitoring approaches primarily capture event occurrence and provide limited insight into procedural quality, timing, and individualized feedback. To address these limitations, we present a real-time HH training and assessment framework that combines deep-learning-based WHO step recognition with a protocol-aware decision-support engine deployed on both a Desktop LED display and a Mixed Reality (MR) headset. The system supports two complementary modes: Concurrent-Feedback Coaching (CFC), which provides real-time sequence guidance and corrective prompts, and Uncued Retention Assessment (URA), which evaluates unguided execution and summarizes detected steps and errors. To support real-time deployment, we retrained and evaluated YOLOv12+MV, TimeSformer, and TSM on four heterogeneous HH datasets (PSCUH, Jurmala, METC, and Kaggle). While several YOLOv12 variants achieved strong recognition performance, the compact YOLOv12-n+MV model provided the most favorable accuracy-efficiency trade-off for deployment, achieving F1-scores of 0.99, 0.87, 0.72, and 0.58 across Kaggle, Jurmala, METC, and PSCUH, respectively, with low computational cost. This lightweight recognizer was integrated with temporal majority voting and a protocol-aware controller to support stable closed-loop interaction on Desktop and HoloLens 2. We evaluated the framework in a controlled mixed-methods study with $N=20$N=20 participants using a $2\times 2$2×2 design (Desktop vs. MR; CFC vs. URA). Desktop yielded significantly faster, more temporally stable, and less error-prone HH performance than MR, whereas CFC reduced total completion time and URA reduced weighted mistake scores, indicating a speed-accuracy trade-off. Subjective results showed higher perceived usability for Desktop than for MR and for CFC than for URA. NASA-TLX further showed a higher workload for MR than Desktop across five subscales under counterbalancing, while URA increased perceived effort relative to CFC. Overall, these findings suggest that Desktop is more suitable when efficiency, stability, and lower workload are priorities, whereas CFC and URA can be selectively used to emphasize guided acquisition or independent recall within Desktop and MR HH training workflows.