Event-based Optical Flow Using Spatio-temporal Registration

Optical flow forms a fundamental information for various motion related vision problems: e.g., SLAM, visual odometry, and object motion estimation. Event cameras are ideal vision sensors for on-line, dynamic tasks that require optical flow estimation, as they have high temporal resolution, high dynamic range and low latency. However, efficiently decoding optical flow from events for high frequency operation while maintaining accuracy is still an open problem. Batch-based optical flow algorithms (CNN or contrast maximisation) accumulate event data over a short period of time and achieve state-of-the-art performance in terms of accuracy, but at the cost of algorithm latency and lower update rates (on par with traditional cameras). In contrast, event-by-event algorithms only compute flow vectors in small, local regions, achieving a lower latency, but losing accuracy when global information is ignored. In this paper, we introduce a spatio-temporal registration framework to increase accuracy of current state-of-the-art event-by-event flow estimation, while also introducing a twofold algorithm acceleration approach and a real-time implementation strategy to mitigate the impact of computation scaling with event rate. We evaluate our event by-event optical flow algorithm on MVSEC, achieving state-of-the-art results for event-by-event algorithms, and performance comparable to batch-based methods. Our method is also computationally efficient, enabling processing of the higher resolution DSEC dataset, and is the only event-by-event algorithm tested to run completely in real time. Furthermore, we demonstrate its effectiveness and efficiency through qualitative evaluations on the ECD and the high-resolution M3ED datasets. Finally, we introduce a moving object dataset, which is outside the autonomous driving domain, to evaluate the general applicability of the proposed optical flow algorithm. The code is available open-source [CODE AVAILABLE ON ACCEPTANCE].