A Latent Diffusion for Stable Frame Interpolation

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Video frame interpolation (VFI) is crucial for enhancing video quality and smoothness.
• Existing VFI methods struggle with maintaining spatiotemporal consistency and visual fidelity.
• Latent diffusion models (LDMs) show promise but require specialized adaptation for VFI tasks.

Purpose of the Study:

• To develop a novel VFI method, LD4SFI, that leverages LDMs for improved frame generation.
• To enhance spatiotemporal smoothness and visual quality in interpolated video sequences.
• To achieve state-of-the-art performance on standard VFI benchmarks.

Main Methods:

• LD4SFI utilizes a latent diffusion model (LDM) integrated with a vector-quantized spatiotemporal variational autoencoder (VQ-STVAE).
• The method conditions the diffusion model on disparity maps to explicitly incorporate motion dynamics and spatiotemporal differences.
• This approach captures intrinsic relationships in spatiotemporal data and integrates complementary information across frames.

Main Results:

• LD4SFI achieved competitive or superior accuracy compared to state-of-the-art (SOTA) methods on UCF-101, DAVIS, and SNU-FILM datasets.
• Quantitative results include low LPIPS and FID scores, and high PSNR and SSIM values, indicating superior visual quality.
• The method demonstrated the highest throughput among diffusion models on a V100 GPU.

Conclusions:

• LD4SFI offers a robust and efficient approach to video frame interpolation.
• The method significantly enhances video continuity and visual quality, outperforming existing SOTA techniques.
• LD4SFI represents a significant advancement in diffusion model applications for video processing.