In-sensor compressing via programmable optoelectronic sensors based on van der Waals heterostructures for intelligent machine vision

Haoxin Huang ¹, Shuhui Shi ^2,3, Jiajia Zha ¹

¹Department of Electrical Engineering, City University of Hong Kong, Hong Kong SAR, China.
²Department of Electrical and Electronic Engineering, University of Hong Kong, Hong Kong SAR, China.
³School of Microelectronics, Southern University of Science and Technology, Shenzhen, 518055, China.
⁴College of Physics and Optoelectronic Engineering, Shenzhen University, Shenzhen, 518060, China.
⁵College of Integrated Circuits and Optoelectronic Chips, Shenzhen Technology University, Shenzhen, 518118, China.
⁶Department of Chemistry, The Chinese University of Hong Kong, Hong Kong SAR, China.
⁷Department of Electronic & Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China.
⁸Department of Materials Science and Engineering, City University of Hong Kong, Hong Kong SAR, China.
⁹Department of Chemistry, City University of Hong Kong, Hong Kong SAR, China.
¹⁰Department of Applied Physics, The Hong Kong Polytechnic University, Hong Kong SAR, China. ychai@polyu.edu.hk.
¹¹School of Microelectronics, Southern University of Science and Technology, Shenzhen, 518055, China. wangzr@sustech.edu.cn.
¹²Department of Electrical Engineering, City University of Hong Kong, Hong Kong SAR, China. chaoltan@cityu.edu.hk.
¹³Department of Biomedical Engineering, City University of Hong Kong, Hong Kong SAR, China. chaoltan@cityu.edu.hk.
¹⁴Hong Kong Branch of National Precious Metals Material Engineering Research Center (NPMM), City University of Hong Kong, Kowloon, Hong Kong SAR, 999077, China. chaoltan@cityu.edu.hk.

⁰Department of Electrical Engineering, City University of Hong Kong, Hong Kong SAR, China.

Nature Communications +

|

April 23, 2025

View abstract on PubMed

Summary

This study introduces a novel two-dimensional (2D) optoelectronic sensor for in-sensor data compression in intelligent machine vision. The sensor achieves an 8:1 compression ratio for dynamic videos and spectral data, enabling efficient edge computing.

Area Of Science

Optoelectronics
Materials Science
Computer Vision

Background

Intelligent machine vision requires efficient capture of multidimensional signals.
In-sensor computing offers reduced data transfer but limited temporal/spectral data compression.
Existing methods struggle with efficient compression of complex visual data.

Purpose Of The Study

To demonstrate a programmable 2D heterostructure-based optoelectronic sensor for in-sensor data compression.
To integrate sensing, memory, and computation within a single device.
To evaluate the sensor's performance in compressing dynamic videos and spectral data.

Main Methods

Development of a programmable two-dimensional (2D) heterostructure-based optoelectronic sensor.
Integration of sensing, memory, and computation functionalities.
In-device snapshot compression of dynamic videos and 3D spectral data.
Evaluation of reconstruction quality using peak signal-to-noise ratio (PSNR).
Classification of compressed video data using in-sensor convolution.

Main Results

Achieved an 8:1 compression ratio for dynamic videos and 3D spectral data.
Demonstrated comparable reconstruction quality (15.81 dB) to software methods (16.21 dB).
Preserved semantic information in compressed videos for accurate classification.
Attained high classification accuracy (93.18%) for compressed videos via in-sensor convolution.

Conclusions

The developed 2D optoelectronic sensor enables efficient in-device data compression.
This technology supports the development of edge-based intelligent vision systems.
The sensor's capabilities advance efficient processing of spectral and temporal visual information.