SETJiP: Spatial and Extra Temporal Jigsaw Puzzles for Video Anomaly Detection

Video Anomaly Detection (VAD) is commonly formulated as a one-class classification task. Global motion, with temporal variations across most pixels within an object-centric region, e.g., walking, is typically regular, whereas localized motion, e.g., waving, can be ambiguous. Decoupled spatial and temporal jigsaw puzzles (DSTJiP) is a self-supervised method that learns discriminative representations by predicting the original order of spatially and temporally shuffled patches. However, DSTJiP's uniform sampling and equal weighting do not assign stronger supervision to global-motion examples within the temporal objective. Consequently, the temporal supervision allocated to global-motion examples may become insufficient across training-data regimes with varying proportions of these examples, deteriorating VAD performance. Nevertheless, excessively strengthening such supervision also degrades performance. To address these issues, we propose spatial and extra temporal jigsaw puzzles (SETJiP) with two RGB-only training schemes that provide stronger and more conservative temporal supervision for global-motion examples, respectively. One scheme strengthens temporal supervision on these examples via additional temporal jigsaw puzzles. The other does so more conservatively by upweighting their temporal jigsaw puzzles. Experiments on four VAD benchmarks show that both schemes improve on DSTJiP and remain highly competitive with state-of-the-art methods.