CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Model.

Area of Science:

• Computer Vision
• Natural Language Processing
• Artificial Intelligence

Background:

• Vision-language models (VLMs) are foundational for many tasks, yet scene text recognition (STR) predominantly uses single-modality pre-trained models.
• VLMs like CLIP demonstrate robust capabilities in identifying diverse text types, including regular and irregular formats.
• Adapting VLMs for STR offers potential for improved performance beyond traditional visual-only approaches.

Purpose of the Study:

• To introduce CLIP4STR, a novel scene text recognition method utilizing the CLIP vision-language model.
• To develop a method that effectively integrates visual and textual semantics for enhanced text recognition.
• To establish a strong baseline for future STR research leveraging VLMs.

Main Methods:

• CLIP4STR employs a dual encoder-decoder architecture with separate visual and cross-modal branches.
• A visual branch provides initial text predictions based on image features.
• A cross-modal branch refines predictions by reconciling visual features with text semantics using a dual predict-and-refine decoding scheme.

Main Results:

• CLIP4STR achieved state-of-the-art performance across 13 scene text recognition benchmarks.
• Scaling the model size, pre-training, and training data significantly boosted CLIP4STR's effectiveness.
• Empirical studies provide insights into CLIP's adaptation for scene text recognition tasks.

Conclusions:

• CLIP4STR demonstrates the efficacy of adapting VLMs, specifically CLIP, for scene text recognition.
• The proposed dual-branch architecture and decoding scheme effectively leverage multi-modal information.
• CLIP4STR serves as a robust and simple baseline for advancing VLM-based scene text recognition research.