VimTS: A Unified Video and Image Text Spotter for Enhancing the Cross-Domain Generalization

Area of Science:

• Computer Vision
• Machine Learning
• Natural Language Processing

Background:

• Text spotting, extracting text from images/videos, struggles with cross-domain generalization.
• Existing methods often require extensive data and parameters for adaptation.

Purpose of the Study:

• To enhance the cross-domain generalization ability of text spotting models.
• To develop a method that effectively adapts single-task models to multi-task image and video scenarios with minimal parameters.

Main Methods:

• Introduced VimTS, a method utilizing a Prompt Queries Generation Module and a Tasks-aware Adapter.
• Developed a synthetic video text dataset (VTD-368 k) using the Content Deformation Fields (CoDeF) algorithm.
• Converted single-task models into multi-task models for improved synergy.

Main Results:

• VimTS achieved an average improvement of 2.6% over state-of-the-art methods in six cross-domain benchmarks.
• For video-level cross-domain adaptation, VimTS surpassed previous methods by 5.5% on the MOTA metric using only image-level data.
• Demonstrated superior performance compared to Large Multimodal Models in cross-domain scene text spotting with fewer parameters and data.

Conclusions:

• VimTS significantly enhances cross-domain text spotting generalization for both image and video data.
• The proposed method offers an efficient approach to multi-task learning for text spotting, requiring fewer parameters and data.
• VimTS presents a more effective alternative to existing Large Multimodal Models for cross-domain scene text spotting challenges.