An image and legend text alignment model based on position guided enhanced attention mechanism

Area of Science:

• Computer Science
• Artificial Intelligence
• Document Image Analysis

Background:

• Understanding document image structure is vital for information extraction.
• Current methods struggle with fine-grained semantic matching between image regions and legend texts.
• Existing models primarily focus on overall layout analysis, lacking detailed image-text correlation.

Purpose of the Study:

• To develop an effective model for fine-grained matching between image regions and legend texts in document images.
• To enhance the understanding of document image structure through accurate image-text relationship identification.
• To address the limitations of current methods in detailed semantic alignment.

Main Methods:

• A fusion model utilizing a position-guided enhanced attention mechanism.
• Separate processing of image and legend text features using heterogeneous structures.
• Integration of region coordinates as auxiliary features for local structure perception.
• Application of a cross-attention mechanism for deep feature fusion between modalities.

Main Results:

• Achieved matching accuracy rates of 90.34% on the Ancient Book Digitization Dataset.
• Attained a matching accuracy rate of 68.5% on the DocBank Dataset.
• Outperformed existing mainstream methods in fine-grained structural understanding tasks.

Conclusions:

• The proposed fusion model demonstrates superior performance in fine-grained image-text matching.
• The position-guided enhanced attention and cross-attention mechanisms are effective for detailed document structure analysis.
• This approach significantly advances the field of document image understanding and information extraction.