Ultrafast visual perception beyond human capabilities enabled by motion analysis using synaptic transistors

Area of Science:

• Robotics and Neuromorphic Engineering
• Biologically Inspired Computing
• Computer Vision Hardware

Background:

• Current optical flow algorithms, while effective on benchmarks, exhibit time delays hindering real-world robotic applications.
• Biological visual systems offer insights into efficient motion processing, particularly the retina-lateral geniculate nucleus (LGN) pathway.
• Need for hardware solutions that can process temporal motion cues rapidly for dynamic environments.

Purpose of the Study:

• To develop a neuromorphic temporal-attention hardware for direct, hardware-based extraction of temporal motion cues.
• To overcome the limitations of conventional optical flow algorithms by reducing processing time delays.
• To enhance robotic capabilities in dynamic environments through ultrafast motion analysis.

Main Methods:

• Emulation of the retina-LGN interaction using a two-dimensional synaptic transistor array.
• Encoding brightness changes and accumulating them in analog, non-volatile states to generate regions of interest (ROIs).
• Utilizing ROIs as inputs for downstream optical flow and vision algorithms.

Main Results:

• Demonstrated ultrafast motion analysis with a 400% speedup compared to state-of-the-art algorithms.
• Achieved performance surpassing human-level speed while maintaining or improving accuracy.
• Synaptic transistor hardware exhibits high-frequency response (~100 μs), non-volatility (>10,000 s), and high endurance (>8,000 cycles).

Conclusions:

• The developed neuromorphic temporal-attention hardware significantly accelerates motion analysis for robotics.
• This hardware-based approach offers a viable solution for real-time robotic perception in dynamic scenarios.
• The system leverages temporal priors for enhanced accuracy and speed, paving the way for advanced robotic vision.