Real-Time High-Definition Hyperspectral Endoscopy via Spatial-Temporal Low-Frequency-Stochastic Spectral Encoding

Hyperspectral endoscopy enables minimally invasive visualization of both structural and compositional information, offering promising potential for the accurate assessment of in vivo physiological and pathological conditions. However, current hyperspectral endoscopy suffers from low frame rate, hindering the clear capture of in vivo tissues in motion, restricting in vivo diagnostics, efficacy assessment, or risk monitoring during minimally invasive procedures. Here we propose a hyperspectral endoscopy by developing a spatial-temporal spectral encoding approach based on low-frequency stochastic filters combined with an encoding-guided spectral attention network (ESANet) to reconstruct the hyperspectral image with low latency. A prototype system is developed to achieve real-time frame rate (20 Hz), high-definition resolution (full pixels), 67 spectral channels spanning 420-750 nm. It can overcome the continuous motion of in vivo tissue to provide hyperspectral images with fine superficial features, including capillary as small as around 37 µm in diameter, reveal the distinct spectra characteristics for diverse types of organs, and enable visualization of rapid and subtle compositional changes in two representative processes: photodynamic therapy and hepatic ischemia. With minimal hardware modifications, the proposed scheme provides a cost-efficient and easily adaptable solution for hyperspectral endoscopy as well as broader application scenarios.