Hierarchical Set-to-Set Representation for 3-D Cross-Modal Retrieval

Three-dimensional in-domain retrieval has recently achieved significant success, but 3-D cross-modal retrieval still faces problems and challenges. Existing methods only rely on a simple global feature (GF), which overlooks the local information of complex 3-D objects and the connections between similar local features across complex multimodal instances. To tackle this issue, we propose a hierarchical set-to-set representation (HSR) and a corresponding hierarchical similarity that incorporates global-to-global and local-to-local similarity metrics. Specifically, we employ feature extractors for each modality to learn both GFs and local feature sets. We then project these features into their respective common space and use bilinear pooling to generate compact-set features that maintain the invariant for set-to-set similarity measurement. To facilitate effective hierarchical similarity measurement, we design an operation to combine the GF and the compact-set feature to generate the hierarchical representation for 3-D cross-modal retrieval, which preserves hierarchical similarity measurement. To optimize the framework, we adopt the joint loss functions, including cross-modal center loss (CMCL), mean square loss, and cross-entropy loss, to reduce the cross-modal discrepancy for each instance and minimize the distances between the instances in the same category. Experimental results demonstrate that our method outperforms the state-of-the-art methods on the 3-D cross-modal retrieval task on both ModelNet10 and ModelNet40 datasets.