NSX-Net: A Neurolinguistic and Acoustic Multimodal Deep Learning Framework for Speech Disorder Classification

Area of Science:

• Computational linguistics and speech processing.
• Artificial intelligence in healthcare.
• Multimodal machine learning for biomedical applications.

Background:

• Speech disorders necessitate advanced multimodal analysis for accurate assessment.
• Current deep learning methods face challenges in temporal synchronization and feature learning.
• Existing frameworks exhibit limitations in interpretability and robustness for diverse speech impairments.

Purpose of the Study:

• To introduce NSX-Net, an explainable multimodal deep learning framework for speech disorder classification.
• To enhance the accuracy and robustness of computer-assisted speech disorder assessment.
• To address limitations in temporal synchronization, feature learning, and interpretability of existing methods.

Main Methods:

• NSX-Net utilizes acoustic speech signals and neurolinguistic text data for classification.
• The Adaptive Speech Refinement Module (ASRM) preprocesses and normalizes multimodal data.
• Hierarchical Multimodal Feature Learning Unit (HMFLU) extracts local and global speech features.
• Dual-Path Attention Enhancement Block (DPAEB) and Temporal Resolution Synchronization Module (TRSM) improve feature representation and temporal consistency.

Main Results:

• NSX-Net achieved superior performance on six public datasets.
• Key performance metrics include 99.27% Accuracy, 99.11% Precision, 98.97% Recall, and 99.02% F1-Score.
• The framework significantly outperformed existing state-of-the-art methods in speech disorder classification.

Conclusions:

• NSX-Net offers an effective and interpretable solution for multiclass speech disorder prediction.
• The framework demonstrates significant potential for clinical applicability in computer-assisted diagnosis.
• Optimized multimodal representations contribute to improved accuracy and robustness in speech disorder assessment.