From Visual to Multimodal: Systematic Ablation of Encoders and Fusion Strategies in Animal Identification

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Automated animal identification systems struggle with limited data and single-cue reliance.
• Current methods for reuniting lost pets face challenges in accuracy and scalability.

Purpose of the Study:

• To develop a multimodal verification framework for enhanced animal identification.
• To improve pet re-identification by integrating visual features with semantic identity priors.

Main Methods:

• Constructed a large training corpus of 1.9 million animal photographs.
• Utilized SigLIP2-Giant (vision) and E5-Small-v2 (text) as optimal backbones.
• Evaluated various fusion strategies, including gated fusion, for multimodal integration.

Main Results:

• Achieved Top-1 accuracy of 84.28% and an Equal Error Rate of 0.0422.
• Demonstrated an 11% improvement over unimodal baselines.
• Showcased the effectiveness of integrating synthesized semantic descriptions.

Conclusions:

• Multimodal verification significantly refines decision boundaries in large-scale pet re-identification.
• The proposed framework offers a practical solution for reuniting lost pets.
• Semantic priors derived from text enhance the performance of visual identification systems.