SketchTrans: Disentangled Prototype Learning With Transformer for Sketch-Photo Recognition

Area of Science:

• Computer Vision
• Machine Learning

Background:

• Sketch-photo recognition faces information asymmetry due to modality differences.
• Existing methods like CNNs and GANs have limitations in discriminability and generation noise.

Purpose of the Study:

• To develop an information-aligned sketch transformer (SketchTrans+) for improved sketch-photo recognition.
• To address the challenge of information asymmetry between sketch and photo modalities.

Main Methods:

• Designed an asymmetric disentanglement scheme with a dynamic auxiliary sketch (A-sketch) to align modality representations.
• Employed a modality-aware prototype contrastive learning method to mine shared information.
• Utilized a transformer architecture for discriminative visual modeling.

Main Results:

• The proposed SketchTrans+ method demonstrates superior performance on category- and instance-level sketch-based datasets.
• The asymmetric disentanglement effectively transfers sketch-irrelevant knowledge to compensate for missing sketch information.
• Modality-aware prototypes enhance the mining of representative modality-sharing information.

Conclusions:

• SketchTrans+ offers a novel and effective approach to sketch-photo recognition by tackling information asymmetry.
• The cross-modal disentangled prototype learning framework significantly improves matching accuracy and discriminability.