Leveraging Transformer-GNN Integration for Multilingual News Speech-to-Text Similarity Modeling

Area of Science:

• Natural Language Processing
• Speech Recognition
• Machine Learning

Background:

• Growing volume of multilingual news necessitates advanced cross-lingual speech-to-text analysis.
• Existing methods struggle with linguistic diversity, ambiguity, and cross-lingual semantic alignment.
• Need for robust systems to transform multilingual speech into comparable text.

Purpose of the Study:

• To develop an integrated Transformer-Graph Neural Network (GNN) framework for multilingual news speech-to-text similarity modeling.
• To enable accurate transformation of multilingual speech into semantically comparable text.
• To overcome limitations of traditional speech-to-text and textual similarity methods.

Main Methods:

• Utilized a Transformer encoder for deep contextual speech embeddings.
• Structured embeddings into graphs to represent semantic relations.
• Employed a GNN to model cross-lingual relational dependencies.
• Implemented a cross-lingual semantic alignment module for similarity scoring.

Main Results:

• Framework outperformed baseline models on multilingual news datasets (English, Hindi, Marathi, Tamil).
• Achieved significant improvements: 7.8% in semantic similarity accuracy, 6.1% in BLEU score, 8.4% in cross-lingual alignment efficiency.
• Demonstrated robustness to noisy input, code-switching, and low-resource scenarios.
• Relative improvement of 4.8% in semantic similarity and 3.1% reduction in word error rate.

Conclusions:

• The Transformer-GNN framework offers a superior solution for multilingual speech-to-text similarity.
• The approach is effective for practical multilingual news applications, even with challenging inputs.
• Future work includes real-time deployment, support for more languages, and multimodal data integration.