GSSTU: Generative Spatial Self-Attention Transformer Unit for Enhanced Video Prediction

Area of Science:

• Computer Vision
• Deep Learning
• Video Prediction

Background:

• Future frame prediction is crucial for video generation, autonomous driving, and robotics.
• Traditional methods like recurrent neural networks (RNNs) and convolutional neural networks (CNNs) struggle with long-range dependencies and complex modeling.
• Generative adversarial networks (GANs) show promise but are computationally intensive and unstable.

Purpose of the Study:

• To develop a novel and effective approach for future frame prediction.
• To address limitations of existing methods in capturing temporal dependencies and preserving image quality.

Main Methods:

• A hybrid model combining 3-D Convolutional Neural Networks (CNNs) for encoding and Transformers for sequence modeling.
• Implementation of a spatial self-attention mechanism to capture spatial relationships.
• Introduction of a neighborhood pixel intensity loss to maintain local image details.

Main Results:

• The proposed method achieved superior performance across five public datasets.
• Significant improvements were observed in Structural Similarity Index Measure (SSIM), Peak Signal-to-Noise Ratio (PSNR), and Learned Perceptual Image Patch Similarity (LPIPS) scores.
• Average improvements of 4.64% (SSIM), 18.5% (PSNR), and 42% (LPIPS) over the second-best method were recorded.

Conclusions:

• The novel approach effectively predicts future frames with high fidelity.
• The combination of 3-D CNNs, Transformers, and specialized loss functions enhances prediction accuracy and quality.