Multimodal fall detection for solitary individuals based on audio-video decision fusion processing

Area of Science:

• Computer Science
• Artificial Intelligence
• Signal Processing

Background:

• Falls pose a significant safety risk, particularly for elderly individuals living alone.
• Existing fall detection systems often struggle with environmental variations like poor lighting or limited monitoring range.
• There is a need for robust and reliable fall detection systems that can overcome the limitations of single-modality approaches.

Purpose of the Study:

• To develop and evaluate a multimodal fall detection system integrating audio and video data.
• To enhance the accuracy and reliability of fall detection compared to single-modality methods.
• To provide a non-intrusive and effective solution for detecting falls in real-world environments.

Main Methods:

• A video-based model using YOLOv7-Pose for skeleton joint extraction, followed by a two-stream Spatial Temporal Graph Convolutional Network (ST-GCN) for classification.
• An audio-based model employing log-scaled mel spectrograms and processed through the MobileNetV2 architecture.
• Decision fusion of video and audio models using linear weighting and Dempster-Shafer (D-S) theory.

Main Results:

• The multimodal system significantly outperformed single-modality methods in fall detection.
• Sensitivity increased from 81.67% (video-only) to 96.67% (linear weighting) and 97.50% (D-S theory).
• The fused approach demonstrated enhanced reliability and effectiveness in complex daily life environments.

Conclusions:

• Multimodal fusion of audio and video data offers a superior solution for fall detection.
• The proposed system effectively addresses limitations of traditional video-based methods, improving accuracy and robustness.
• Audio-visual fusion is a promising direction for future fall detection systems, especially for vulnerable populations.