RSRNeT: a novel multi-modal network framework for named entity recognition and relation extraction.

Area of Science:

• Natural Language Processing
• Computer Vision
• Artificial Intelligence

Background:

• Named Entity Recognition (NER) and Relation Extraction (RE) are crucial for knowledge graph construction.
• Uni-modal approaches relying solely on text exhibit limitations in performance and efficiency, especially with polysemous words.
• Existing multi-modal methods for NER (MNER) and RE (MRE) can be inefficient when irrelevant images are present.

Purpose of the Study:

• To propose a novel multi-modal network framework, RSRNeT, for enhanced named entity recognition and relation extraction.
• To improve the extraction of visual features and the comprehensive fusion of multi-modal information.
• To mitigate performance degradation caused by irrelevant images in multi-modal learning tasks.

Main Methods:

• Developed a multi-scale visual feature extraction module using the ResNeSt network.
• Designed a multi-modal feature fusion module based on the RoBERTa network.
• Integrated these modules to learn robust visual and textual representations, minimizing interference from irrelevant visual data.

Main Results:

• RSRNeT demonstrated state-of-the-art performance on MNER and MRE tasks.
• Achieved superior recall and F1 scores compared to baseline models.
• Validated effectiveness on three public datasets: Twitter2015, Twitter2017, and MNRE.

Conclusions:

• The proposed RSRNeT framework effectively enhances multi-modal named entity recognition and relation extraction.
• The novel modules enable superior feature extraction and fusion, leading to improved accuracy and efficiency.
• RSRNeT offers a promising solution for knowledge extraction from multi-modal data, particularly in challenging scenarios with irrelevant information.