Scale-Consistent Fusion: From Heterogeneous Local Sampling to Global Immersive Rendering.

Area of Science:

• Computer Vision
• Computer Graphics
• Multimedia Systems

Background:

• Image-based geometric modeling and novel view synthesis are crucial for virtual reality and immersive telepresence.
• Sparse, large-baseline sampling poses challenges for inferring reliable depth information.
• Commercial light field (LF) cameras offer a convenient way to capture geometry information.

Purpose of the Study:

• To develop a method for rendering high-quality novel views globally using a sparse set of LF captures.
• To address the challenge of scale inconsistency in fusing LF captures from multiple angles.
• To improve disparity inference and LF synthesis for multimedia applications.

Main Methods:

• Proposed a scale-consistent volume rescaling algorithm to align disparity probability volumes (DPV) from different captures.
• Developed attention-guided multi-scale residual fusion and disparity field-guided deep re-regularization modules.
• Fused DPVs projected to the target camera frustum for global geometry fusion.

Main Results:

• The proposed method achieved robust alignment of DPVs, enabling scale-consistent global geometry fusion.
• Learning-based modules effectively regularized noisy observations from heterogeneous captures.
• Significant improvements in disparity inference and LF synthesis were demonstrated on the Stanford Lytro Multi-view LF dataset.

Conclusions:

• The novel scale-consistent volume rescaling algorithm effectively fuses LF captures with varying scales.
• The proposed learning-based modules enhance the quality of novel LFIs rendering.
• The method significantly outperforms existing approaches for disparity inference and LF synthesis.