Hierarchical Set-to-Set Representation for 3-D Cross-Modal Retrieval

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Three-dimensional (3-D) cross-modal retrieval faces challenges due to reliance on global features, neglecting local object details and inter-modal feature connections.
• Existing methods fail to capture the intricate relationships between local features across different modalities of complex 3-D objects.

Purpose of the Study:

• To propose a novel hierarchical set-to-set representation (HSR) for enhanced 3-D cross-modal retrieval.
• To address the limitations of global feature-only approaches by incorporating local feature information and cross-modal similarities.

Main Methods:

• Developed a hierarchical set-to-set representation (HSR) incorporating global-to-global and local-to-local similarity metrics.
• Utilized feature extractors to learn global features (GFs) and local feature sets, projecting them into a common space with bilinear pooling for compact-set features.
• Designed a hierarchical similarity measurement combining GFs and compact-set features, optimized using joint loss functions (CMCL, mean square loss, cross-entropy loss).

Main Results:

• The proposed HSR method significantly outperforms state-of-the-art approaches in 3-D cross-modal retrieval.
• Experimental validation on ModelNet10 and ModelNet40 datasets confirms the effectiveness of the hierarchical representation and similarity metrics.

Conclusions:

• The novel HSR framework effectively captures both global and local information, leading to superior performance in 3-D cross-modal retrieval.
• This approach offers a more robust solution for retrieving 3-D objects across different modalities by preserving hierarchical similarities.