Eliminating Data Duplication in CQA Platforms Using Deep Neural Model.

Area of Science:

• Natural Language Processing
• Artificial Intelligence
• Machine Learning

Background:

• Existing duplicate question detection methods primarily use English datasets.
• Community Question Answering (CQA) platforms often feature informal, bilingual user queries.
• There is a need for robust methods to handle code-mixed languages like Hinglish.

Purpose of the Study:

• To develop and validate a deep learning model for detecting semantically identical, duplicate questions in transliterated bilingual data.
• To address the challenge of duplicate question detection in informal languages like Hinglish within CQA systems.

Main Methods:

• A two-module hybrid deep learning model was proposed.
• Module 1: Language transliteration to convert bilingual input into mono-lingual text.
• Module 2: A hybrid model combining Siamese neural networks, capsule networks, and a decision tree classifier for similarity computation using Manhattan distance.

Main Results:

• The model achieved an accuracy of 87.0885% on a dataset of 150 question pairs.
• An Area Under the Receiver Operating Characteristic Curve (AUC-ROC) value of 0.86 was obtained.
• The model successfully identified duplicate questions in transliterated Hinglish data.

Conclusions:

• The proposed hybrid deep learning model effectively detects duplicate questions in transliterated bilingual data, including informal Hinglish.
• This approach enhances the efficiency and accuracy of information retrieval in CQA systems dealing with multilingual content.