A Lightweight Radar-Camera Fusion Deep Learning Model for Human Activity Recognition.

Area of Science:

• Computer Vision
• Machine Learning
• Sensor Fusion

Background:

• Privacy-preserving human activity recognition is crucial for indoor environments.
• Existing methods struggle with illumination variations and background clutter.
• Anonymity must be maintained during sensing.

Purpose of the Study:

• To propose a lightweight deep learning model fusing radar and camera data for human activity recognition.
• To achieve robust and privacy-preserving activity recognition.
• To enable efficient deployment on edge devices.

Main Methods:

• Utilized Frequency Modulated Continuous Wave (FMCW) radar for motion signatures and ultra-low-resolution cameras for spatial cues.
• Processed radar data as Range-Doppler-Time cubes and used Transformer models for temporal analysis.
• Employed a privacy-preserving 4x5-pixel camera input with difference frames and a dedicated Transformer encoder.
• Fused modality-specific feature vectors and classified activities using a lightweight fully connected network.

Main Results:

• Achieved 98.74% classification accuracy on a multimodal dataset with 15 activity classes.
• Significantly outperformed single-modality radar and camera baselines.
• The model demonstrated high efficiency with only 11 million floating-point operations (11 MFLOPs).

Conclusions:

• The proposed radar-camera fusion model offers a highly accurate and efficient solution for privacy-preserving human activity recognition.
• The lightweight design makes it suitable for deployment on resource-constrained embedded or edge devices.
• This approach effectively integrates micro-Doppler patterns and coarse spatial cues for robust activity detection.