Space-Time Video Super-Resolution With Neural Operator

Area of Science:

• Computer Vision
• Artificial Intelligence
• Signal Processing

Background:

• Space-time video super-resolution (STVSR) aims to enhance video resolution across both spatial and temporal dimensions.
• Current STVSR methods struggle with accurate motion estimation and compensation (MEMC) for significant object movements.
• Physics-informed neural networks offer a new paradigm for modeling complex physical processes.

Purpose of the Study:

• To develop an advanced STVSR method that overcomes limitations in MEMC for large motions.
• To leverage continuous function spaces and physics-informed principles for improved spatiotemporal detail reconstruction.
• To introduce an efficient and accurate MEMC mechanism for enhanced video super-resolution.

Main Methods:

• Modeling MEMC challenges in STVSR as a mapping between continuous function spaces.
• Transforming low-resolution representations into high-resolution ones with enriched spatiotemporal details.
• Designing a Galerkin-type attention function for efficient frame alignment and temporal interpolation.

Main Results:

• The proposed method achieves superior performance compared to state-of-the-art techniques in STVSR.
• Demonstrated effectiveness in both fixed-size and continuous STVSR tasks.
• The Galerkin-type attention mechanism provides linear complexity and global receptive fields for precise large motion estimation.

Conclusions:

• The novel physics-informed approach significantly advances STVSR capabilities, particularly for scenarios with large motions.
• The developed Galerkin-type attention mechanism offers an efficient and accurate solution for MEMC.
• The method sets a new benchmark for STVSR performance, with code publicly available.