Weakly Supervised Learning for Textbook Question Answering.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Computer Vision

Background:

• Textbook Question Answering (TQA) requires deep text comprehension and effective diagram understanding for multi-modal contexts.
• Existing methods struggle with the specificity of integrating textual and visual information.

Purpose of the Study:

• To propose a novel Weakly Supervised learning method for TQA (WSTQ) that enhances text understanding and diagram semantics.
• To leverage intermediate task results for self-supervision and improve overall TQA performance.

Main Methods:

• WSTQ utilizes Text Matching (TM) and Relation Detection (RD) tasks, trained with weakly supervised signals.
• Text understanding is pre-trained on TM and fine-tuned on TQA.
• Diagram semantics are learned via RD, focusing on relationships between detected regions, and trained jointly with TQA using multitask learning.

Main Results:

• WSTQ achieved significant accuracy improvements of 5.02% on CK12-QA and 4.12% on AI2D.
• The method demonstrates effective learning of both text comprehension and diagram semantics.
• Experimental validation confirms the superiority over state-of-the-art baselines.

Conclusions:

• The proposed Weakly Supervised learning approach (WSTQ) effectively addresses TQA challenges by improving text and diagram understanding.
• Multitask learning of TM and RD tasks enhances the model's ability to interpret complex multi-modal educational content.
• The released code facilitates further research in multi-modal question answering.