Synchronous Analysis of Speech Production and Lips Movement to Detect Parkinson's Disease Using Deep Learning Methods

Area of Science:

• Biomedical Engineering
• Computational Linguistics
• Neurology

Background:

• Parkinson's disease (PD) impacts over 6 million globally, necessitating accurate diagnosis and monitoring.
• Current automated methods for PD detection often rely on isolated speech or facial video analysis.
• Existing approaches struggle to fully leverage the complementary information present in both speech and visual cues.

Purpose of the Study:

• To introduce a novel bimodal methodology for Parkinson's disease detection.
• To perform synchronous fusion of speech signals and lip movement videos for enhanced PD pattern modeling.
• To evaluate the efficacy of this bimodal approach against unimodal and asynchronous methods.

Main Methods:

• Developed a new methodology for synchronous fusion of speech and lip movement data.
• Implemented a bimodal approach integrating information from audio and visual (lip movement) recordings.
• Utilized attention mechanisms for synchronous speech-to-lips and lips-to-speech information fusion.

Main Results:

• The synchronous bimodal approach demonstrated superior accuracy and suitability over unimodal methods.
• The proposed method outperformed classical asynchronous fusion techniques that neglect temporal information.
• Results confirmed the complementary nature of lip movement and speech production when using advanced fusion strategies.

Conclusions:

• Synchronous fusion using attention-based concatenated projections significantly improves Parkinson's disease classification.
• Multimodal approaches combining visual and speech signals offer robust and confident models for clinical diagnostic support.
• This study highlights the potential of advanced multimodal fusion for improving PD diagnosis and patient management.