Video Reenactment as Inductive Bias for Content-Motion Disentanglement

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Disentangled representations are crucial for downstream tasks and data explanation.
• Video-based disentangled factors of variation offer valuable low-dimensional representations for task-specific models.

Purpose of the Study:

• To introduce MTC-VAE, a self-supervised motion-transfer VAE, for disentangling motion and content from videos.
• To achieve temporal consistency in per-chunk representations and enable whole video reconstruction in a single pass.

Main Methods:

• Utilizes a chunk-wise modeling approach, leveraging spatiotemporal neighborhood motion information.
• Extends the ELBO's log-likelihood with a Blind Reenactment Loss to encourage motion disentanglement.
• Assumes that swapping motion features enables reenactment between videos.

Main Results:

• Outperforms existing methods on video motion-content disentanglement metrics.
• Demonstrates effective disentanglement for video reenactment tasks.
• Achieves superior reconstruction quality and motion alignment compared to baselines.

Conclusions:

• The MTC-VAE model successfully disentangles motion and content in videos.
• The proposed Blind Reenactment Loss effectively leverages motion disentanglement.
• The model shows significant improvements in video reenactment and reconstruction quality.