Bayesian selective scan for robust long-horizon action recognition via reliability-aware discretization

Long-horizon action recognition is increasingly deployed in streaming and safety-critical settings, where short but severe corruptions, such as motion blur or adversarial perturbations, can overwrite temporal memory and cause errors to persist after unreliable segments end. Selective State Space Models (SSMs) enable linear-time inference over long contexts, but deterministic discretization can amplify this memory-contamination failure mode.

We propose BayesMamba, a reliability-aware selective SSM that integrates Bayesian uncertainty into SSM discretization. BayesMamba uses Bayesian Selective Scan (B-Scan) to estimate per-frame aleatoric and epistemic uncertainty, convert uncertainty into a reliability signal, and modulate both the effective time step and input injection so that state updates are suppressed when observations are unreliable. We evaluate BayesMamba on Kinetics-400 (K400) and Something-Something V2 (SSv2).

Under clean evaluations, BayesMamba achieves comparable or slightly improved performance, with 82.4% top-1 accuracy on K400 and 68.7% on SSv2, compared with 82.1% and 68.2% for a deterministic selective SSM baseline, while retaining similar computational cost. On SSv2 under bursty motion blur, BayesMamba improves robust accuracy from 51.4% to 54.2%; under projected gradient descent (PGD, ε = 4/255, ε = 8/255, and ε = 16/255), it improves robust accuracy from 18.2% to 19.8%. BayesMamba also reduces post-corruption persistence, decreasing Persist(64) from 0.41 to 0.19 and Persist(128) from 0.38 to 0.16. Under distribution shift, it improves calibration, reducing corrupted-set ECE from 0.158 to 0.097 and corrupted-set NLL from 2.34 to 1.66.

These results indicate that reliability-aware discretization mitigates memory contamination and improves long-horizon robustness with modest overhead while preserving clean action recognition accuracy.