MLNet: a multi-level multimodal named entity recognition architecture.

Area of Science:

• Human-Computer Interaction
• Artificial Intelligence
• Natural Language Processing
• Computer Vision

Background:

• Accurate identification of talking objects is crucial for robot decision-making and recommendations.
• Named Entity Recognition (NER) and Object Detection (OD) are fundamental for object recognition in NLP and CV.
• Existing multimodal approaches show promise but require optimization for noisy, short text-image data in Multimodal Named Entity Recognition (MNER).

Purpose of the Study:

• To propose a new multi-level multimodal named entity recognition architecture.
• To enhance semantic understanding and entity identification efficacy in MNER tasks.
• To address limitations in current image-text-based MNER architectures, particularly with noisy data.

Main Methods:

• Separate image and text encoding.
• A symmetric Transformer-based neural network for multimodal feature fusion.
• A gating mechanism to filter relevant visual information and enhance text understanding.
• Character-level vector encoding to mitigate text noise.
• Conditional Random Fields for label classification.

Main Results:

• The proposed model demonstrated increased accuracy in the MNER task on the Twitter dataset.
• Effective filtering of visual information improved semantic disambiguation.
• Character-level encoding reduced the impact of text noise on recognition accuracy.

Conclusions:

• The novel multi-level multimodal architecture significantly improves MNER task performance.
• The gating mechanism and character-level encoding are effective strategies for handling noisy multimodal data.
• This work advances the capabilities of robots in understanding and interacting with their environment through improved object recognition.