Related Experiment Video
Updated: Jul 16, 2025
06:32
Author Spotlight: Automated Deep Brain Stimulation for Parkinson's Disease - Exploring the Possibilities and Challenges of Home Monitoring
Published on: July 14, 2023
1.3K
Real-time Context-Aware Multimodal Network for Activity and Activity-Stage Recognition from Team Communication in
Chenyang Gao1, Ivan Marsic1, Aleksandra Sarcevic2
1Rutgers University, United States.
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
|September 18, 2023
Summary
This study introduces a novel speech-based system for recognizing activities and their stages during trauma resuscitation. The multimodal approach enhances clinical event monitoring without relying on visual data or high-performing automatic speech recognition.
Area of Science:
- Clinical informatics
- Speech processing
- Medical artificial intelligence
Background:
- Current clinical activity recognition systems primarily rely on visual or sensory data, limiting their application in scenarios lacking these cues.
- Activities involving verbal assessment or complex team communication during critical events, such as trauma resuscitation, are challenging for existing recognition technologies.
Purpose of the Study:
- To develop and evaluate a speech-based system for recognizing activities and activity stages in clinical settings, specifically during trauma resuscitation.
- To address the limitations of visual-dependent systems by focusing on audio signals and team communication patterns.
Main Methods:
- Collected a high-quality dataset of common activities and stages during actual trauma resuscitation events.
- Developed a novel multimodal network utilizing audio signals and keywords, independent of a high-performance automatic speech recognition (ASR) engine.
- Designed contextual modules to capture dynamic dependencies in team conversations and introduced a data augmentation method simulating team communication.
Main Results:
- The context-aware multimodal model achieved offline F1-scores of 73.2±0.8% for activity recognition and 78.1±1.1% for activity-stage recognition.
- Online experiments showed performance declines of approximately 10-15% depending on ASR output segmentation, demonstrating robustness.
- The data augmentation method improved performance in a data-limited scenario.
Conclusions:
- Speech-based activity and activity-stage recognition is feasible during dynamic clinical events like trauma resuscitation.
- The proposed multimodal approach offers a viable alternative for monitoring clinical activities where visual or device data is unavailable.
- Further research can optimize ASR integration and segmentation for enhanced real-time performance.
