Unambiguous Scene Text Segmentation with Referring Expression Comprehension.

Area of Science:

• Computer Vision
• Natural Language Processing
• Artificial Intelligence

Background:

• Understanding natural scenes is crucial for real-world applications.
• Existing visual phrase grounding methods primarily focus on general objects.
• Accurate scene text segmentation from descriptions remains a challenge.

Purpose of the Study:

• To develop a novel framework for scene text segmentation using referring expressions.
• To enable precise segmentation of text instances from complex backgrounds.
• To establish a new dataset for quantitative evaluation of scene text referring expression segmentation.

Main Methods:

• A unified deep network jointly models visual and linguistic information.
• Encoding region-level and pixel-level visual features into spatial feature maps.
• Decoding feature maps into a saliency response map for text instances.

Main Results:

• The proposed framework achieves effective text instance segmentation.
• Experimental results demonstrate superior performance on the COCO-CharRef dataset.
• The method outperforms baselines from state-of-the-art text localization and retrieval methods.

Conclusions:

• Combining image-based visual features with language-based textual explanations enhances scene text segmentation.
• The novel framework provides an effective solution for segmenting scene text instances based on referring expressions.
• The developed COCO-CharRef dataset facilitates further research in this area.