Virtual View Synthesis for Free Viewpoint Video and Multiview Video Compression using Gaussian Mixture Modelling.

Area of Science:

• Computer Vision
• Image Processing
• 3D Graphics

Background:

• Virtual view synthesis is crucial for immersive 3D experiences in free viewpoint video and multiview video coding (MVC).
• Existing methods struggle with rendering quality due to occlusions and warping errors, particularly at foreground-background boundaries.
• Limitations include holes in virtual views and quality degradation from low spatial correlation and inter-view depth map correspondence.

Purpose of the Study:

• To develop an advanced view synthesis technique that overcomes limitations of current methods.
• To improve the quality of synthesized virtual views by addressing holes and boundary artifacts.
• To enhance the realism of 3D viewing experiences in video coding applications.

Main Methods:

• Utilized Gaussian mixture modeling (GMM) to effectively separate foreground and background pixels.
• Implemented an adaptive weighted average for recovering missing pixels based on GMM models and warped image data.
• Introduced an adaptive GMM model resetting strategy to handle dynamic scene changes and object motion.

Main Results:

• Achieved significant Peak Signal-to-Noise Ratio (PSNR) improvements of 5.40-6.60 dB over existing view synthesis methods.
• Demonstrated effectiveness when integrated as an extra reference frame in MVC motion estimation, yielding 3.15-5.13 dB PSNR gains.
• The proposed technique successfully reduces artifacts and enhances the visual quality of synthesized virtual views.

Conclusions:

• The GMM-based view synthesis technique offers a robust solution for high-quality virtual view generation.
• The method effectively handles occlusions and boundary regions, improving 3D video rendering.
• This approach significantly advances the state-of-the-art in multiview video coding and free viewpoint video applications.