M$^{3}$3D: A Multimodal, Multilingual and Multitask Dataset for Grounded Document-Level Information Extraction.

Area of Science:

• Natural Language Processing
• Computer Vision
• Multimodal AI

Background:

• Multimodal information extraction (IE) benefits text analysis, but existing datasets lack video-based IE and fine-grained visual grounding.
• Current resources primarily focus on sentence-level, image-facilitated IE in English.

Purpose of the Study:

• To address limitations in existing multimodal IE datasets, this study introduces M³3D, a multimodal multilingual multitask dataset.
• The dataset supports document-level text and video, English and Chinese languages, and diverse IE tasks including entity recognition, relation extraction, and visual grounding.
• It also introduces the biography domain to enrich multimodal IE resources.

Main Methods:

• A novel hierarchical multimodal IE model was developed to leverage and integrate multimodal information.
• The Denoised Feature Fusion Module (DFFM) was designed for effective multimodal information integration.
• A Missing Modality Construction Module (MMCM) was introduced to handle incomplete modal information in non-ideal scenarios.

Main Results:

• The proposed model achieved an average performance of 53.80% (English) and 53.77% (Chinese) across four tasks.
• These results establish a benchmark for subsequent research in multimodal IE.
• Analytical experiments validated the effectiveness of the proposed DFFM and MMCM modules.

Conclusions:

• The M³3D dataset and the proposed IE model advance the field of multimodal information extraction.
• This work promotes the development of more robust and versatile multimodal IE systems.
• The dataset's multilingual and multitask nature facilitates broader research and application of multimodal IE.