Alignment Relation is What You Need for Diagram Parsing.

Area of Science:

• Computer Science
• Artificial Intelligence
• Data Science

Background:

• Diagrams are crucial knowledge carriers but challenging for AI due to sparse visual elements and ambiguous semantics.
• Existing methods struggle to accurately capture the precise semantics of visual elements in diagrams.

Purpose of the Study:

• To develop a method for assigning precise semantics to visual elements in diagrams by aligning them with textual elements.
• To introduce the first dataset, Align Diagram Element (ADE), for visual-textual element alignment in diagrams.
• To propose a novel Visual-Textual Alignment Model (VTAM) for enhanced diagram understanding.

Main Methods:

• Constructing relational graphs between visual and textual elements using four relational operators (distance, connection, inclusion, feature similarity).
• Implementing an optimal aligning phase where element representations are refined through a weighted sum across relational graphs.
• Building the Align Diagram Element (ADE) dataset with annotations for visual-textual element alignment relations.

Main Results:

• The Visual-Textual Alignment Model (VTAM) achieved a 10.9% improvement on the ADE dataset compared to the previous best method.
• Integrating VTAM into diagram-related tasks like Diagram Question Answering (DQA) resulted in significant performance gains.
• Improvements of 2.8%–5.9% on AI2D and 4.6%–5.1% on Foodwebs were observed after incorporating VTAM.

Conclusions:

• The proposed VTAM effectively addresses the challenge of precise semantic interpretation in diagrams.
• Alignment relations are crucial for advancing diagram parsing and related AI applications.
• The ADE dataset and VTAM offer valuable resources for future research in diagram understanding.