Hypergraph-Based Multi-Modal Representation for Open-Set 3D Object Retrieval | JoVE Visualize

Area of Science:

Computer Vision and Machine Learning
3D Data Analysis and Retrieval

Background:

Traditional 3D object retrieval (3DOR) operates under a close-set assumption, limiting its real-world applicability due to unseen categories.
Existing methods often fail to learn generalized 3D object embeddings, hindering performance when encountering novel object classes.

Purpose of the Study:

To introduce and address the open-set 3D object retrieval task, expanding the scope beyond traditional close-set limitations.
To propose a novel framework, Hypergraph-Based Multi-Modal Representation (HGM^2R), for learning robust 3D object embeddings in an open-set setting.

Main Methods:

The HGM^2R framework comprises two modules: Multi-Modal 3D Object Embedding (MM3DOE) for unified embeddings from diverse data modalities (multi-view, point clouds, voxels) and Structure-Aware and Invariant Knowledge Learning (SAIKL) for high-order correlations.
SAIKL utilizes hypergraph modeling to capture complex relationships among objects and a memory bank for aligning embeddings with typical representations, enhancing generalization.
Formal proof demonstrates the superior representational capability of hypergraph modeling over traditional graph modeling for data correlation.

Main Results:

Four new multi-modal datasets (OS-ESB-core, OS-NTU-core, OS-MN40-core, OS-ABO-core) were generated for evaluating open-set 3DOR.
The proposed HGM^2R method significantly outperforms existing approaches, achieving state-of-the-art results with notable improvements (e.g., 12.12%/12.88% mAP gains on specific datasets).
Experimental results and visualizations confirm the framework's effectiveness in extracting generalized 3D object embeddings for open-set retrieval.

Conclusions:

The HGM^2R framework provides a powerful solution for the challenging open-set 3D object retrieval task.
The method demonstrates strong generalization capabilities, effectively handling unseen object categories through multi-modal fusion and hypergraph-based learning.