Bridging the latency gap with a continuous stream evaluation framework in event-driven perception

Area of Science:

• Computer Vision
• Neuromorphic Engineering

Background:

• Neuromorphic vision systems process event streams for real-time applications.
• Current evaluation methods use RGB imaging, ignoring perception latency, creating a performance gap.

Purpose of the Study:

• Introduce a novel evaluation framework, STream-based lAtency-awaRe Evaluation (STARE), to address latency issues in neuromorphic vision.
• Quantify the impact of latency on online accuracy and propose solutions.

Main Methods:

• Developed the STARE framework with Continuous Sampling and Latency-Aware Evaluation.
• Created ESOT500, a high-dynamic object tracking dataset with 500 Hz annotations for validation.
• Introduced Asynchronous Tracking and Context-Aware Sampling for model enhancement.

Main Results:

• Latency was found to degrade online accuracy by over 50%.
• The STARE framework effectively quantifies latency-induced accuracy loss.
• Proposed enhancement strategies improved model throughput and handling of low event density.

Conclusions:

• The STARE framework bridges the gap between theoretical potential and real-world deployment of neuromorphic vision systems.
• Addressing latency is crucial for accurate and reliable real-time performance.
• New evaluation and enhancement methods pave the way for more robust neuromorphic applications.