Evaluating the Performance of Sensor-based Bout Detection Algorithms: The Transition Pairing Method

Bout detection algorithms are used to segment data from wearable sensors, but it is challenging to assess segmentation correctness.

To present and demonstrate the Transition Pairing Method (TPM), a new method for evaluating the performance of bout detection algorithms.

The TPM compares predicted transitions to a criterion measure in terms of number and timing. A true positive is defined as a predicted transition that corresponds with one criterion transition in a mutually exclusive pair. The pairs are established using an extended Gale-Shapley algorithm, and the user specifies a maximum allowable within-pair time lag, above which pairs cannot be formed. Unpaired predictions and criteria are false positives and false negatives, respectively. The demonstration used raw acceleration data from 88 youth who wore ActiGraph GT9X monitors (right hip and non-dominant wrist) during simulated free-living. Youth Sojourn bout detection algorithms were applied (one for each attachment site), and the TPM was used to compare predicted bout transitions to the criterion measure (direct observation). Performance metrics were calculated for each participant, and hip-versus-wrist means were compared using paired T-tests (α = 0.05).

When the maximum allowable lag was 1-s, both algorithms had recall <20% (2.4% difference from one another, p<0.01) and precision <10% (1.4% difference from one another, p<0.001). That is, >80% of criterion transitions were undetected, and >90% of predicted transitions were false positives.

The TPM improves on conventional analyses by providing specific information about bout detection in a standardized way that applies to any bout detection algorithm.