On the Effect of Training Convolution Neural Network for Millimeter-Wave Radar-Based Hand Gesture Recognition

Area of Science:

• Computer Vision
• Machine Learning
• Human-Computer Interaction

Background:

• Millimeter-wave radar-based hand gesture recognition (MR-HGR) faces challenges with small training datasets.
• Transfer learning from computer vision models offers a potential solution.

Purpose of the Study:

• To investigate the effectiveness of fine-tuning state-of-the-art Convolutional Neural Networks (CNNs) for MR-HGR.
• To propose a novel data representation, the temporal space-velocity (TSV) spectrogram, for radar echo signals.

Main Methods:

• Fine-tuning various CNN architectures (ResNet, DenseNet, MobileNet V2, ShuffleNet V2) using radar data.
• Utilizing TSV spectrograms as an integrated data modality for hand gesture features.
• Conducting both self-testing (ST) and cross-testing (CT) to evaluate model generalization.
• Performing an auxiliary test with augmented data to simulate adaptation to new users.

Main Results:

• Cross-testing (CT) achieved an average accuracy exceeding 93%, with self-testing (ST) nearing 100%.
• Augmenting training data with challenging gestures improved average accuracy from ~55-65% to over 90% for new users.
• The proposed TSV spectrogram effectively captures time-evolving hand gesture features.

Conclusions:

• Fine-tuning pre-trained CNNs is a viable strategy to overcome data scarcity in MR-HGR.
• The TSV spectrogram representation enhances feature extraction for radar-based gestures.
• The approach demonstrates significant potential for improving MR-HGR in real-world applications and consumer electronics.