Neural illumination calibration for surgical workflow-optimized spectral imaging

Hyperspectral imaging (HSI) is emerging as a promising novel imaging modality with various potential surgical applications. Currently available cameras, however, suffer from poor integration into the clinical workflow because they require the lights to be switched off or the camera to be manually recalibrated as soon as lighting conditions change.

We propose a novel learning-based approach to recalibration of hyperspectral cameras during surgery that predicts the corresponding white reference image from an uncalibrated hyperspectral input, enabling spatially resolved, automatic, and sterile calibration under varying illumination conditions. Our key novelty lies in (i) the disentanglement of the space of possible illuminations from the space of possible tissue configurations and (ii) combining real-world white reference measurements with physics-inspired simulated illuminations to create a diverse and representative training set.

Based on a total of 1,890 HSI cubes from a phantom, porcine subjects, rats, and humans, we derive the following key insights: Firstly, dynamically changing lighting conditions in the operating room dramatically reduce the performance of methods for physiological parameter estimation and surgical scene segmentation. Secondly, our method is not only sufficiently accurate to replace the tedious process of white reference-based recalibration, but also outperforms previously proposed methods by a large margin. Finally, our approach generalizes across species, lighting conditions, and image processing tasks.

Our method enables seamless integration of hyperspectral imaging into surgical workflows by providing rapid and automated illumination calibration. Its robust generalization across diverse conditions significantly enhances the reliability and practicality of spectral imaging in clinical settings, paving the way for broader adoption of HSI in surgery.