End-to-End Implementation of a Convolutional Neural Network on a 3D-Integrated Image Sensor with Macropixel Array.

Area of Science:

• Computer Vision
• Sensor Technology
• Artificial Intelligence

Background:

• Three-dimensional-integrated focal-plane array image processors enable in-sensor computer vision.
• Neural networks are crucial for complex machine vision tasks but require efficient implementation.
• Previous studies focused on implementing only parts of neural networks, like convolutional kernels, on pixel processors.

Purpose of the Study:

• To implement a complete, continuous convolutional neural network pipeline directly on a macropixel processor array chip.
• To demonstrate efficient, end-to-end machine vision inference within an image sensor.

Main Methods:

• Developed a continuous pipeline from photon digitization to output prediction.
• Utilized a macropixel processor array chip where processors act on groups of pixels.
• Exploited multi-level parallelism within the sensor for computation.

Main Results:

• Achieved a processing rate of 265 to 309 frames per second.
• Successfully implemented an end-to-end convolutional neural network pipeline.
• Performed inference directly inside the image sensor, eliminating external processing.

Conclusions:

• Demonstrated the feasibility of full convolutional neural network inference within image sensors.
• Highlighted the potential of integrated sensor-processor chips for high-speed computer vision.
• Showcased the benefits of exploiting sensor-level parallelism for efficient machine vision.