Integrating Cross-Modal Semantic Learning with Generative Models for Gesture Recognition.

Area of Science:

• Ubiquitous Computing
• Human-Computer Interaction
• Signal Processing

Background:

• Radio frequency (RF)-based WiFi sensing offers low-cost gesture recognition for ubiquitous computing.
• Current methods face challenges like manual data collection, multipath interference, and poor cross-domain generalization.
• Existing data augmentation techniques often overlook the biomechanical structure inherent in RF signals.

Purpose of the Study:

• To develop a cross-modal gesture recognition framework (CM-GR) that integrates semantic learning and generative modeling.
• To address limitations in WiFi sensing by incorporating biomechanical constraints and enabling user-specific data generation.
• To improve the accuracy and scalability of WiFi-based gesture recognition.

Main Methods:

• Leveraging 3D skeletal points from vision data as semantic priors to guide realistic WiFi signal synthesis.
• Incorporating biomechanical constraints into WiFi signal generation without extensive manual labeling.
• Utilizing dynamic conditional vectors derived from inter-subject skeletal differences for personalized WiFi data generation.

Main Results:

• CM-GR significantly enhances cross-subject gesture recognition accuracy.
• Achieved accuracy gains of up to 10.26% on the MM-Fi dataset and 9.5% on the SelfSet dataset.
• Demonstrated the framework's effectiveness in synthesizing personalized WiFi data.

Conclusions:

• CM-GR effectively synthesizes personalized WiFi data by integrating biomechanical information.
• The proposed method overcomes limitations of manual data collection and improves generalization.
• CM-GR shows strong potential for robust and scalable gesture recognition in practical ubiquitous computing settings.