Neural Sensors: Learning Pixel Exposures for HDR Imaging and Video Compressive Sensing With Programmable Sensors.

Area of Science:

• Computational Photography
• Sensor Technology
• Machine Learning in Imaging

Background:

• Traditional camera sensors use fixed global or rolling shutters, limiting capture of high-dynamic-range (HDR) scenes and high-speed events.
• Existing spatially varying pixel exposure methods require heuristic coding and bulky spatial light modulators.
• A need exists for advanced computational photography techniques to overcome current sensor limitations.

Purpose of the Study:

• To introduce neural sensors, a novel methodology for optimizing per-pixel shutter functions.
• To enable end-to-end optimization of shutter functions jointly with differentiable image processing methods like neural networks.
• To demonstrate implementation of optimized exposure functions directly on re-configurable sensor-processors.

Main Methods:

• Developed a neural sensor methodology optimizing per-pixel shutter functions.
• Integrated shutter function optimization with differentiable image processing (neural networks) for end-to-end learning.
• Leveraged programmable and re-configurable sensor-processors for on-sensor implementation of optimized exposure functions.

Main Results:

• Optimized physically feasible optical codes by considering specific sensor limitations.
• Demonstrated successful application of neural sensors for snapshot HDR imaging.
• Validated performance for high-speed compressive imaging through simulations and experimental evaluations with real scenes.

Conclusions:

• Neural sensors offer a powerful, integrated approach to computational photography.
• This methodology overcomes limitations of fixed shutter functions for advanced imaging tasks.
• The system enables efficient, on-sensor implementation of optimized exposure strategies for improved image capture.