MINIGE-MNER: A multi-stage interaction network inspired by gene editing for multimodal named entity recognition

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Computer Vision

Background:

• Multimodal Named Entity Recognition (MNER) combines text and image data for entity identification.
• Existing MNER methods struggle with imbalanced modality noise, semantic mismatch, and information loss due to lack of text dominance.

Purpose of the Study:

• To propose a novel Multi-stage Interaction Network Inspired by Gene Editing for MNER (MINIGE-MNER).
• To address the limitations of existing MNER techniques by improving noise handling, semantic alignment, and text dominance.

Main Methods:

• A gene knockout module using variational information bottleneck for balanced modality noise filtering.
• A gene recombination site determination module to enhance cross-modal semantic alignment.
• A text-guided gene recombination module for a text-dominant, vision-supplementary fusion paradigm.

Main Results:

• MINIGE-MNER achieved F1 scores of 76.45% on Twitter-2015 and 88.67% on Twitter-2017.
• Outperformed state-of-the-art methods by 0.83% and 0.42% on the respective datasets.
• Comprehensive experiments validated the superiority and module effectiveness of MINIGE-MNER.

Conclusions:

• MINIGE-MNER effectively mitigates modality noise, semantic mismatch, and information loss in MNER.
• The proposed gene editing-inspired network offers a significant advancement in multimodal information fusion for entity recognition.