Cascaded Dynamic Memory Refinement and Semantic Alignment for Exo-to-Ego Cross-View Video Generation

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Cross-view video generation between exocentric and egocentric perspectives is challenging due to viewpoint discrepancies and limited feature overlap.
• Existing methods struggle with long-range temporal context and incorporating egocentric semantic information, impacting synthesis quality.

Purpose of the Study:

• To develop a cue-free video generation approach that effectively addresses the challenges of cross-view synthesis.
• To enhance the temporal context modeling and semantic understanding for generating realistic egocentric videos from exocentric inputs.

Main Methods:

• Proposed a cascaded Dynamic memory Refinement and Semantic Alignment (DRSA) framework.
• Implemented Dynamic Memory Refinement (DMR) using a dynamic memory and cross-attention transformer for long-range temporal dynamics.
• Introduced Viewpoint-aware Semantic Alignment (VSA) with dual encoder-decoder learning to bridge the semantic gap between views.

Main Results:

• The DRSA method demonstrated superior performance in cross-view video generation compared to state-of-the-art techniques.
• Quantitative metrics and qualitative evaluations confirmed the effectiveness of the proposed approach.
• A new dataset with dynamic scenes and interacting objects was created to support future research.

Conclusions:

• The proposed DRSA method significantly improves cross-view video generation by effectively integrating temporal information and semantic alignment.
• The novel components, DMR and VSA, successfully address limitations in previous approaches.
• The new dataset facilitates further advancements in egocentric video synthesis research.