Simpler is Better: Feature Guard and Interaction for Semantic Correspondence

Semantic correspondence establishes keypoint correspondences between different instances of the same category. Fusing texture and semantic features from vision foundation models like stable diffusion (SD) and DINO significantly improves matching performance. However, we found an unnoticed yet essential problem: current feature fusion enhances the edge and semantic information in SD features with fine textures and DINOv2 features with fine semantics, but it destroys the semantic and structural information in SD features with weak and coarse semantics. We propose guard features (GuFT), a simple yet efficient method, to prevent feature degradation. Moreover, matching methods designed for traditional deep neural networks can be simplified based on two key insights: 1) vision foundation models provide rich visual knowledge; and 2) GuFT yields high-quality feature descriptors. We propose a bottleneck-style non-shared aggregation and backward interaction (NABI) module to efficiently capture intra- and inter-feature relationships, instead of common self- and cross-attention. The resulting framework, SimBetter, embodies a "simpler is better" design philosophy. It achieves state-of-the-art results with lower computation on SPair-71k, AP-10K, and PF-PASCAL, excelling in geometry-aware, cross-species, cross-family, and cross-dataset tasks. SimBetter also shows excellent potential in the applications of image-video semantic correspondence and sticker editing. Code is available at https://github.com/wzhlearning/SimBetter.