Action Recognition and Benchmark Using Event Cameras.

Area of Science:

• Computer Vision
• Machine Learning
• Robotics

Background:

• Traditional action recognition relies on frame-based cameras, which have limitations.
• Event cameras offer advantages by capturing only brightness changes, but research in event-based action recognition is limited.
• Large-scale datasets for event-based action recognition are scarce.

Purpose of the Study:

• To propose a novel framework for event-based action recognition.
• To introduce a large-scale benchmark dataset for event-based action recognition.
• To enhance the practicality and efficiency of event-based action recognition systems.

Main Methods:

• Proposed the EV-ACT framework incorporating a Learnable Multi-Fused Representation (LMFR) for integrating multiple event information.
• Utilized an event-based slow-fast network with dual temporal granularity for fusing appearance and motion features.
• Introduced a spatial-temporal attention mechanism to improve action recognition capabilities.

Main Results:

• Collected and released THU^E-ACT-50 and THU^E-ACT-50-CHL, the largest event-based action recognition datasets to date, featuring over 12,830 recordings across 50 categories.
• Achieved significant performance improvements over previous methods, with gains of 14.5%, 7.6%, 11.2%, and 7.4% on four benchmarks.
• Demonstrated the practicality and efficiency of the EV-ACT framework through deployment on a mobile platform.

Conclusions:

• The proposed EV-ACT framework effectively addresses the challenges in event-based action recognition.
• The new large-scale datasets will significantly advance research in this field.
• EV-ACT shows promise for real-world applications due to its efficiency and performance.