A Multimodal Convolutional Neural Network Model for Parkinson's Disease Diagnosis Based on Fused Handwriting Dynamics...

Area of Science:

• Neurology and Artificial Intelligence
• Biomedical Signal Processing
• Machine Learning for Healthcare

Background:

• Parkinson's disease (PD) diagnosis requires precision and robustness, with early detection crucial for management.
• Handwriting dynamics show promise as a biomarker for early PD detection.
• Current diagnostic methods for PD often lack sufficient accuracy and reliability.

Purpose of the Study:

• To develop a novel multimodal deep learning decision support system for enhanced early Parkinson's disease diagnosis.
• To integrate static and dynamic handwriting features using advanced signal processing techniques.
• To improve diagnostic accuracy and interpretability in Parkinson's disease detection.

Main Methods:

• A multimodal deep learning model combining static handwriting images with fused time-frequency representations (STFT, CWT) of grip pressure, axial pressure, tilt, and accelerometer data.
• Utilized Short-Time Fourier Transform (STFT) and Continuous Wavelet Transform (CWT) to generate spectrograms and scalograms from sensor signals.
• Employed Gradient-weighted Class Activation Mapping++ (Grad-CAM++) for explainable AI (XAI) to ensure model interpretability.

Main Results:

• Fusing STFT spectrograms achieved 85.41% accuracy, improving to 97.92% with the multimodal CNN.
• Fusing CWT scalograms yielded 92.08% accuracy, enhanced to 96.66% with the multimodal approach.
• The CWT-based approach outperformed STFT, and integrating fused time-frequency images with visualizations further boosted accuracy.

Conclusions:

• Fused time-frequency representations of handwriting dynamics are effective for Parkinson's disease diagnosis.
• The multimodal deep learning approach, particularly with CWT, offers high-precision and explainable PD detection.
• Further validation on diverse datasets is needed to confirm generalizability and clinical applicability.