In-sensor analog optoelectronic processing of concurrent event and memory signals for dynamic vision sensing

Area of Science:

• Neuromorphic Engineering
• Computer Vision
• Sensor Technology

Background:

• Efficient dynamic vision processing requires capturing rapid changes and temporal context.
• Current image and event sensors are power-intensive due to digital processing.

Purpose of the Study:

• To develop an in-sensor architecture for concurrent analog event spike and persistent memory tail generation.
• To enable ultralow-latency and ultralow-power neuromorphic vision.

Main Methods:

• Integration of phosphor pairs with silicon photodiodes and transimpedance amplifiers.
• Development of a dual-response architecture with microsecond and millisecond kinetics.
• Utilizing a convolutional neural network for action and trajectory classification.

Main Results:

• Prototype sensor successfully reconstructed event frames and captured motion history.
• Achieved high accuracy in human action (93.1%) and vehicle trajectory (98.0%) classification.
• Demonstrated accurate speed estimation (2.15 km/h error) and improved action classification (93.3%) with a compressive optical neural network front end.

Conclusions:

• The proposed in-sensor dual-response architecture significantly reduces power consumption and latency.
• Eliminates the need for analog-to-digital conversion and digital accumulation for enhanced vision processing.
• Paves the way for efficient, real-time dynamic vision in AI systems.