Instruction-Guided Scene Text Recognition.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Natural Language Processing

Background:

• Multi-modal models excel in visual recognition but struggle with scene text recognition (STR) due to compositional differences.
• Existing STR models face challenges in understanding the nuances of text within natural scenes.

Purpose of the Study:

• To propose a novel instruction-guided scene text recognition (IGTR) paradigm.
• To adapt instruction learning for STR by focusing on character attribute prediction.
• To develop a flexible and efficient STR system.

Main Methods:

• Formulating STR as an instruction learning problem.
• Utilizing instruction triplets to describe character attributes (e.g., frequency, position).
• Developing a lightweight instruction encoder, cross-modal feature fusion, and multi-task answer head for attribute learning.

Main Results:

• IGTR significantly outperforms existing STR models on English and Chinese benchmarks.
• The model maintains a small size and fast inference speed.
• IGTR effectively addresses challenges in recognizing rare and morphologically similar characters.

Conclusions:

• The proposed IGTR paradigm offers a new character-understanding-based approach to STR.
• IGTR demonstrates superior performance and efficiency compared to current methods.
• The instruction-guided approach provides flexibility and improved handling of difficult character recognition cases.