A hierarchical multimodal system for motion analysis in patients with epilepsy

Area of Science:

• Neuroscience and Neurology
• Computer Science and Artificial Intelligence

Background:

• Seizure semiology, the analysis of clinical signs during seizures, is crucial for understanding involved cerebral networks and for presurgical evaluation in drug-resistant epilepsy.
• Distinguishing between mesial temporal lobe epilepsy (MTLE) and extratemporal lobe epilepsy (ETLE) relies on semiological patterns, but manual analysis is time-consuming and requires expertise.
• Automated analysis of seizure semiology, particularly across multiple modalities, faces challenges due to clinical variables like patient covering and inadequate lighting.

Purpose of the Study:

• To develop and evaluate a novel modular, hierarchical, multimodal system for automated detection and quantification of semiologic signs from 2D monitoring videos.
• To leverage computer vision and deep learning to jointly learn semiologic features from facial, body, and hand motions for improved epilepsy localization.
• To quantitatively classify epilepsy types based on detected semiology, aiming to enhance diagnostic precision and support presurgical assessments.

Main Methods:

• Development of a hierarchical, multimodal system integrating computer vision and deep learning architectures.
• Joint learning of semiologic features from facial, body, and hand movements captured in 2D videos.
• Validation using a dataset of 161 seizures from an Australian epilepsy unit, with leave-one-subject-out cross-validation.

Main Results:

• The system achieved classification accuracies for semiological patterns from the face, body, and hands ranging between 12%–83.4%, 41.2%–80.1%, and 32.8%–69.3%, respectively.
• Demonstrated the potential of multimodal analysis in capturing complex semiologic progressions reflective of neuronal network integration.
• The system provides a quantitative approach to semiology, overcoming limitations of manual, time-intensive clinical assessments.

Conclusions:

• The proposed hierarchical multimodal system represents a significant advancement towards fully automated semiology analysis in epilepsy.
• This technology has the potential to improve the diagnostic precision of epilepsy localization and aid in surgical decision-making.
• Further development could lead to more objective and efficient presurgical evaluations for patients with drug-resistant epilepsy.