Toward in-sensor imaging classification enabled by on-chip all-optical modulation and photonic neural networks

Image sensors play a crucial role in autonomous driving and security. However, the separation of sensors and processors results in large amounts of redundant data being exchanged between sensory terminals and computing units. Here, leveraging silicon photonic integrated technology, we propose an in-sensor imaging classification solution that combines sensing components with photonic neural networks. The sensing components consist of a microring array which can convert visible light signals from free space into near-infrared light signals propagating through silicon waveguides, which are subsequently processed by photonic neural networks. The proposed all-optical modulator is based on a pn-doped microring resonator, enabling light-to-light modulation through two mechanisms: the thermo-optic and plasma-dispersion effects. We demonstrate that these effects can be controlled by shorting the pn-junctions, achieving a modulation depth of 15 dB. Two cascaded microrings-used for converting visible light signals and applying weight signals-demonstrate the ability to perform dot product operations with an effective resolution of near 8 bits, sufficient for data processing. Furthermore, we validated the in-sensor scheme using the Modified National Institute of Standards and Technology (MNIST) dataset, achieving a recognition accuracy of 96.82%. These results pave the way for advanced 'in-sensor' imaging classification technologies.