Turning a CLIP Model Into a Scene Text Spotter.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Natural Language Processing

Background:

• Scene text detection and spotting are crucial for various AI applications.
• Existing methods often struggle with diverse text appearances and limited training data.
• Large-scale pre-trained models offer potential for enhanced performance.

Purpose of the Study:

• To develop a robust backbone for scene text detection and spotting using Contrastive Language-Image Pretraining (CLIP).
• To enhance the synergy between visual and textual information for refined text region identification.
• To improve performance, inference speed, and few-shot learning capabilities in text recognition tasks.

Main Methods:

• Utilized CLIP's visual prompt learning and cross-attention for feature extraction.
• Introduced an instance-language matching process with predefined and learnable prompts.
• Developed a Bimodal Similarity Matching (BSM) module for dynamic language prompt generation.
• Integrated FastTCM-CR50 as a backbone for existing text detection and spotting models.

Main Results:

• Achieved average performance improvements of 1.6% in text detection and 1.5% in text spotting when enhancing existing models.
• Outperformed the TCM-CR50 backbone with average gains of 0.2% and 0.55% in detection and spotting, respectively, while increasing inference speed by 47.1%.
• Demonstrated robust few-shot learning, improving performance by 26.5% and 4.7% with only 10% of supervised data.
• Showcased consistent performance enhancement on out-of-distribution datasets like NightTime-ArT and DOTA.

Conclusions:

• FastTCM-CR50 effectively enhances scene text detection and spotting by leveraging CLIP's capabilities.
• The proposed methods offer significant improvements in performance, speed, and data efficiency.
• The backbone demonstrates strong generalization across various text recognition challenges and datasets.