CLIP-Llama: A New Approach for Scene Text Recognition with a Pre-Trained Vision-Language Model and a Pre-Trained...

Area of Science:

• Computer Vision
• Artificial Intelligence
• Natural Language Processing

Background:

• Scene Text Recognition (STR) is vital for AI applications like image retrieval and intelligent transportation.
• Pre-trained vision-language models are foundational for downstream AI tasks.
• CLIP demonstrates robust text recognition across regular and irregular text formats.

Purpose of the Study:

• To introduce CLIP-Llama, a novel STR model integrating CLIP and Llama2-7B.
• To enhance STR accuracy by combining visual and cross-modal information.
• To establish a strong foundation for future STR research using vision-language models.

Main Methods:

• Utilized CLIP's image and text encoders with two branches: visual and cross-modal.
• Incorporated Llama2-7B in the cross-modal branch for refining predictions.
• Employed a dual prediction and refinement decoding scheme for improved inference.

Main Results:

• CLIP-Llama achieved state-of-the-art performance on 11 Scene Text Recognition benchmark tests.
• Demonstrated robust capabilities in recognizing diverse text in natural images.
• Showcased the effectiveness of the dual-branch architecture and Llama2-7B integration.

Conclusions:

• CLIP-Llama offers a significant advancement in Scene Text Recognition.
• The model's performance highlights the potential of integrating large language models with vision-language models for STR.
• This work provides a solid foundation for future research in AI-driven text recognition.