Implicit hierarchical temporal-spatial residual model for long-term video prediction

Predicting long-term future frames in videos is challenging due to prediction ambiguities and error amplification over time. These issues become more pronounced in distant frames, where small errors accumulate into significant discrepancies. Few studies have addressed hierarchical spatial-temporal representations that effectively manage video stochasticity, particularly across datasets with varying spatial distributions. Given the importance of spatial distribution analysis in computer vision, hierarchical spatial modeling has been shown to outperform many non-autoregressive likelihood-based models, particularly in video spatial analysis. In this work, we propose a Hierarchical Temporal-Spatial Residual Model for long-term video prediction, which captures the residual distribution between the prior and posterior, facilitating a richer representation of the stochastic features present in videos. Specifically, we propose a hierarchical residual generative model that improves the latent state space's ability to capture spatial features in videos. This approach enhances generalization across diverse spatial distributions in video data. By explicitly modeling the residual nature of the data and aligning the approximate posterior with the prior, our method better captures stochastic variations. As a result, it significantly boosts performance in long-term video prediction tasks. Evaluations on three challenging datasets show that our model outperforms both temporal model-based and convolutional neural network-based approaches. Our code is made publicly available at https://github.com/fengpany/Implicit_video_prediction.