Two-dimensional mesh-based mosaic representation for manipulation of video objects with occlusion

Area of Science:

• Computer Vision
• Image Processing
• Computer Graphics

Background:

• Classical mosaic representations rely on global motion models, limiting their effectiveness with occlusions and deformable objects.
• Handling self-occlusion and object-to-object occlusion remains a challenge in video object representation.

Purpose of the Study:

• To develop a robust 2-D mesh-based mosaic representation for video objects with mildly deformable motion and occlusion.
• To improve video object rendering and editing techniques, including augmented reality applications.

Main Methods:

• Utilizing a 2-D mesh-based mosaic representation with object and mosaic meshes for each frame.
• Employing local affine models to map uncovered regions onto the mosaic reference frame, adapting to occlusions.
• Integrating occlusion-adaptive 2-D mesh tracking for frame-to-frame object mesh propagation and mesh optimization.

Main Results:

• Successful synthesis of 2-D mesh-based mosaics for video objects.
• Demonstrated effectiveness in video object editing applications, including self and synthetic transfiguration.
• Developed an algorithm for determining the minimum still views required for replacement mosaic reconstruction.

Conclusions:

• The proposed 2-D mesh-based mosaic representation effectively handles deformable motion and occlusions in video objects.
• The method offers significant improvements for video object rendering, editing, and augmented reality applications.
• The approach provides a foundation for advanced video manipulation and reconstruction techniques.