ColorSketchNet: Unifying color, sketch and texture for modality-agnostic multi-modal person re-identification

Person re-identification (ReID) using text or sketch as descriptive query has made great efforts in practical application. Existing multi-modal ReID methods usually map features from different modalities into a common space to learn cross-modal alignment. However, they ignore inherent modality attribute differences, resulting in limited shared feature representation. Moreover, unfair modality attributes exacerbate the imbalance in multi-task learning, e.g., Text-RGB, Sketch-RGB, Text and Sketch-RGB. Towards these issues, we propose a ColorSketchNet framework that can adaptively compensate for different modality features and establish a fair attribute space for cross-modal alignment. Specifically, we first introduce a color sketch generator to generate the auxiliary modality that includes rich color and texture sketch, which is utilized to narrow the modality gap with the target image. Second, an attribute compensation and refined module is presented to compensate for the missing information in sketch and text with the help of auxiliary color sketch modality and suppress auxiliary modality noise, thereby enhancing the reliability of shared features across modalities. Further, a unified collaborative alignment learning scheme is developed to guide the model in learning identity representations of each modality. Extensive experiments on three multi-modal ReID datasets show that our ColorSketchNet greatly improves the retrieval accuracy and generalization ability in both paired modality and tri-modality scenarios.