Fast Feature Field (F³): A general predictive representation of events for robot perception

This paper develops a mathematical argument and algorithms for building representations of data from event-based cameras that we call Fast Feature Field (F³). We learned this representation by predicting future events from past events and show that it preserves scene structure and motion information. F³ exploits the sparsity of event data and is robust to noise and variations in event rates. It can be computed efficiently using ideas from multiresolution hash encoding and deep sets, achieving 120 hertz at high-definition (HD) resolution and 440 hertz at video graphics array (VGA) resolution. F³ represents events in a contiguous spatiotemporal volume as a multichannel image, enabling a range of downstream tasks. We obtained state-of-the-art performance on optical flow estimation, semantic segmentation, and monocular metric depth estimation, on data from three robotic platforms (a car, a quadruped robot, and a flying platform) and across different lighting conditions (daytime and nighttime), environments (indoors, outdoors, urban, and off-road), and dynamic vision sensors (resolutions and event rates). Our implementations can predict these tasks at 25 to 75 hertz at HD resolution.