VSFormer: Mining Correlations in Flexible View Set for Multi-View 3D Shape Understanding.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• View-based methods are effective for 3D shape understanding but often rely on rigid assumptions or indirect correlation learning.
• Existing approaches limit flexibility in exploring inter-view relationships, impacting task effectiveness.

Purpose of the Study:

• To investigate flexible organization and explicit correlation learning for multiple views in 3D shape understanding.
• To develop a novel Transformer model that overcomes limitations of current view-based methods.

Main Methods:

• Proposes a permutation-invariant 'View Set' to organize different views of a 3D shape, removing rigid relation assumptions.
• Devises VSFormer, a Transformer model to explicitly capture pairwise and higher-order correlations within the View Set.
• Theoretically establishes a correspondence between Cartesian product of view sets and attention mechanism's correlation matrix.

Main Results:

• VSFormer demonstrates enhanced flexibility and efficient inference.
• Achieves superior performance on various 3D recognition datasets (ModelNet40, ScanObjectNN, RGBD).
• Establishes new state-of-the-art records on the SHREC'17 retrieval benchmark.

Conclusions:

• VSFormer offers a more flexible and effective approach to multi-view 3D shape understanding.
• The proposed method significantly advances the state-of-the-art in 3D recognition and retrieval.
• Explicitly modeling inter-view correlations via a Transformer is a promising direction for 3D computer vision.