TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Large multimodal models (LMMs) are increasingly important for complex tasks.
• Existing LMMs face challenges in text-centric applications, particularly with high-resolution inputs and token efficiency.

Purpose of the Study:

• To introduce TextMonkey, an LMM optimized for text-centric tasks.
• To improve performance and interpretability in document understanding and text spotting.

Main Methods:

• Implemented Shifted Window Attention for enhanced cross-window connectivity and training stability.
• Utilized similarity-based token filtering to reduce redundancy and improve efficiency.
• Integrated text spotting, grounding, and positional information for better interpretability.

Main Results:

• Achieved 5.2% improvement in Scene Text-Centric tasks and 6.9% in Document-Oriented tasks.
• Demonstrated a 10.9% increase in scene text spotting and set a new standard on OCRBench (561 score).
• Outperformed previous open-source LMMs in document understanding benchmarks.

Conclusions:

• TextMonkey offers significant advancements in text-centric LMM capabilities.
• The model's novel techniques lead to superior performance and interpretability.
• TextMonkey establishes a new benchmark for open-source document understanding models.