CNER-Omni: A unified dynamic modality learning framework for Chinese named entity recognition across text and speech

Area of Science:

• Natural Language Processing
• Artificial Intelligence
• Machine Learning

Background:

• Multimodal data is increasingly common in real-world applications.
• Existing Chinese Named Entity Recognition (CNER) methods treat text, speech, and multimodal inputs independently, leading to inefficiencies and poor cross-modal generalization.
• There is a need for a unified framework for Integrated Multimodal Named Entity Recognition (IMNER).

Purpose of the Study:

• To propose CNER-Omni, a unified framework for IMNER that consolidates text-based, speech-based, and multimodal CNER into a single model.
• To introduce IMAGE, an Integrated Multimodal Generation framework that treats NER as an entity-aware sequence generation task.
• To improve cross-modal representation learning and model adaptability to various input configurations.

Main Methods:

• Designed a unified input representation schema for multimodal data.
• Developed IMAGE, an Integrated Multimodal Generation framework utilizing pseudo-modal inputs for cross-modal learning.
• Incorporated a modality-composition-aware mixture-of-experts (MoE) module for dynamic adaptation.
• Conducted experiments on AISHELL-NER, CNERTA, and MSRA benchmarks.

Main Results:

• CNER-Omni achieved state-of-the-art performance across text, speech, and multimodal NER tasks.
• The unified framework significantly reduced model complexity compared to independent approaches.
• Demonstrated strong performance in both flat and nested NER settings.
• Showcased robustness in low-resource and cross-modal scenarios.

Conclusions:

• CNER-Omni provides an effective and efficient unified framework for Integrated Multimodal Named Entity Recognition.
• The IMAGE framework enables superior cross-modal representation learning and adaptability.
• The proposed method sets a new standard for CNER across diverse data modalities and challenging conditions.