Integration of multi-level semantics in PTMs with an attention model for question matching

Area of Science:

• Natural Language Processing (NLP)
• Artificial Intelligence
• Machine Learning

Background:

• Question matching/retrieval is crucial for NLP applications.
• Neural network models achieve high accuracy but struggle with complex cases.
• Leveraging specialized layers in pre-trained language models (PTMs) is key.

Purpose of the Study:

• To propose a novel attention-based model, ERNIE-ATT.
• To effectively integrate diverse semantic levels from PTMs.
• To enhance the robustness of question matching systems.

Main Methods:

• Utilizing specializations encoded in different layers of large-scale PTMs.
• Developing an attention-based mechanism (ERNIE-ATT) for semantic integration.
• Conducting experimental evaluations on challenging datasets.

Main Results:

• ERNIE-ATT significantly outperforms traditional models without PTMs.
• The proposed model shows substantial improvement over existing PTM-based models.
• ERNIE-ATT demonstrates enhanced robustness in question matching.

Conclusions:

• Integrating diverse semantic levels from PTMs via attention is effective.
• ERNIE-ATT offers a robust solution for complex question matching tasks.
• The approach advances the state-of-the-art in semantic equivalence detection.