RoboMNIST: A Multimodal Dataset for Multi-Robot Activity Recognition Using WiFi Sensing, Video, and Audio

Area of Science:

• Robotics
• Artificial Intelligence
• Sensor Networks

Background:

• Multi-robot activity recognition (MRAR) is crucial for advanced autonomous systems.
• Existing methods often require dedicated sensors, increasing deployment costs and complexity.
• Leveraging signals of opportunity, such as WiFi, offers a cost-effective sensing solution.

Purpose of the Study:

• Introduce a novel multimodal dataset for MRAR.
• Integrate WiFi Channel State Information (CSI), video, and audio data for enhanced robotic perception.
• Facilitate the development of robust and accurate MRAR systems.

Main Methods:

• Collected data using two Franka Emika robotic arms.
• Integrated WiFi CSI, video, and audio streams from multiple sensors.
• Utilized signals of opportunity from existing WiFi infrastructure for environmental sensing.

Main Results:

• Developed a comprehensive multimodal dataset for MRAR.
• Demonstrated the potential of combining CSI, visual, and auditory data for improved recognition.
• Enabled a holistic understanding of robotic environments for advanced operations.

Conclusions:

• The novel dataset advances robotic perception and autonomous systems.
• Repurposing WiFi signals for sensing provides a valuable resource for developing sophisticated decision-making capabilities.
• The multimodal approach enhances robustness and accuracy in dynamic environments.