TinyMem: Condensing Multimodal Memory for Long-Form Video Action Detection

Despite the great advances in video understanding with deep neural networks, current solutions still struggle with input videos that last for minutes, if not hours. To mitigate this issue, existing approaches typically build memory caches of dense visual embeddings for video transformers to model long-range spatio-temporal dependencies. However, even with hundreds of extended memory tokens, their performance remains limited. In this paper, we argue that compact yet informative memory representations can effectively improve performance. To this end, we introduce TinyMem, a model built upon compact multimodal memory for long-form video action detection. In particular, we condense redundant video content into succinct descriptions to extract abstract textual semantics. Subsequently, we integrate region-condensed visual embeddings with text embeddings. TinyMem outperforms a range of state-of-the-art models on AVA v2.2, Epic-Kitchens-100 and Breakfast with highly condensed memory, e.g., 37.4 mAP with TinyMem-24-12 on AVA v2.2 while using 5 times fewer memory tokens than the baseline with dense visual memory embeddings.