Multimodal fusion of handwriting images and kinematic features for Parkinson disease detection

Parkinson's disease (PD) affects fine motor control and produces measurable abnormalities in handwriting and drawing. This study proposes a rigorously evaluated multimodal framework for PD detection that combines a Vision Transformer (ViT) for spiral and meander image analysis with an XGBoost classifier operating on 54 carefully engineered kinematic features extracted from multichannel handwriting signals. To assess how multimodal integration should be performed, both intermediate feature-level fusion and late decision-level fusion were evaluated under a strict 5-fold subject-wise cross-validation protocol, with supplementary sample-level analysis, on the NewHandPD dataset. The visual stream consistently outperformed the acquisition stream as a single modality, while both fusion strategies improved performance by exploiting complementary spatial (visual) and motor information. Intermediate fusion achieved the highest apparent discriminative performance, reaching 97.7% accuracy on spiral drawings and 98.5% accuracy on meander drawings, whereas late fusion provided more interpretable and modular behavior, with best subject-level results of 93.94% accuracy and AUC = 0.9687 for spiral, and 92.42% accuracy with AUC = 0.9770 for meander. These findings suggest that multimodal handwriting analysis can be an effective approach for Parkinson's disease detection on the NewHandPD dataset, although further validation on larger and independent cohorts is required.