Use of sensors to quantify procedural idle time: Validity evidence for a new mastery metric

Quantification of mastery is the first step in using objective metrics for teaching. We hypothesized that during orotracheal intubation, top tier performers have less idle time compared to lower tier performers.

At the Anesthesiology 2018 Annual Meeting, 82 participants intubated a normal airway simulator and a burnt airway simulator. The movements of the participant's laryngoscope were quantified using electromagnetic motion sensors. Top tier performers were defined as participants who intubated both simulators successfully in less than the median time for each simulator. Idle time was defined as the duration of time when the laryngoscope was not moving.

Top performers showed less Idle Time when intubating the normal airway compared to lower tier performers (14.5 ± 9.8 seconds vs 34.0 ± 52.0 seconds, respectively P < .01). Likewise, top performers showed less Idle Time when intubating the burnt airway compared to lower tier performers (18.6 ± 15.2 seconds vs 63.4 ± 59.11 seconds; P < .01). Comparing performance on the burnt airway to the normal airway, there was a difference for lower tier performers (63.4 ± 59.1 seconds vs 34.0 ± 52.0 seconds; P < .01) but not for top tier performers (18.6 ± 15.2 seconds vs 14.5 ± 9.8 seconds; P = .07).

Similar to our previous findings with other procedures, Idle Time was shown to have known group validity evidence when comparing top performers with lower tier performers. Further, Idle Time was correlated with procedure difficulty in our prior work. We observed statistically significant differences in Idle Times for lower tier performers when comparing the normal airway to the burnt airway but not for top tier performers. Our findings support the continued exploration of Idle Time for development of objective assessment and curricula.