MTGEA: A Multimodal Two-Stream GNN Framework for Efficient Point Cloud and Skeleton Data Alignment.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Sensor Fusion

Background:

• Human activity recognition is crucial for home care systems, but camera-based methods raise privacy concerns and struggle in low light.
• Radar sensors offer privacy and robust performance in poor lighting but often yield sparse data.

Purpose of the Study:

• To develop a novel framework for efficient point cloud and skeleton data alignment to improve human activity recognition using radar sensors.
• To address the sparsity of radar data and enhance recognition accuracy in home care applications.

Main Methods:

• Collected multimodal datasets using mmWave radar and Kinect v4 sensors.
• Applied data augmentation techniques (zero-padding, Gaussian Noise, Agglomerative Hierarchical Clustering) to increase point cloud density.
• Utilized a Spatial Temporal Graph Convolutional Network (ST-GCN) for spatio-temporal feature extraction.
• Implemented an attention mechanism for aligning multimodal features (point clouds and skeleton data).

Main Results:

• The proposed Multimodal Two-stream GNN Framework for Efficient Point Cloud and Skeleton Data Alignment (MTGEA) significantly improved human activity recognition accuracy.
• The model demonstrated enhanced performance using radar data alone, overcoming previous limitations.
• Accurate skeletal features from Kinect models were leveraged to boost recognition capabilities.

Conclusions:

• The MTGEA framework offers a promising solution for privacy-preserving and accurate human activity recognition in home care.
• The study highlights the potential of fusing radar and skeleton data for robust activity recognition systems.
• The developed model effectively addresses the sparsity of radar data and improves overall recognition performance.