Spectral convolutional neural network chip for in-sensor edge computing of incoherent natural light

Area of Science:

• Optoelectronics
• Artificial Intelligence
• Materials Science

Background:

• Optical neural networks face limitations in integration scale and require coherent light sources.
• Existing optical computing methods are not energy-efficient or practical for real-world applications.

Purpose of the Study:

• To propose and demonstrate a spectral convolutional neural network (SCNN) integrated with matter meta-imaging for in-sensor optical analog computing.
• To utilize incoherent natural light as the information carrier for highly parallel computations.
• To achieve high energy efficiency and scalability for edge computing applications.

Main Methods:

• Implementation of optical convolutional layers using very large-scale, pixel-aligned spectral filters on a CMOS image sensor.
• Development of a spectral convolutional neural network (SCNN) architecture for processing natural light.
• In-sensor analog computing through highly parallel spectral vector-inner products.

Main Results:

• The SCNN chip successfully processed incoherent natural light, enabling in-sensor optical analog computing.
• Achieved over 96% accuracy for pathological diagnosis and nearly 100% accuracy for face anti-spoofing at video rates.
• Demonstrated the feasibility of using the same chip for diverse, complex real-world tasks.

Conclusions:

• This work presents the first integrated optical computing system utilizing natural light.
• The developed SCNN chip offers a feasible and scalable solution for in-sensor edge computing in portable terminals.
• The technology promises high energy efficiency and broad applicability across various domains.