HyKey: hyperspectral keypoint detection and matching in minimally invasive surgery

Area of Science:

• Medical Imaging
• Computer Vision
• Surgical Technology

Background:

• 3D reconstruction in minimally invasive surgery (MIS) is crucial for enhanced surgical guidance, visualization, and tool tracking.
• Traditional RGB-based methods face limitations in surgical scenes due to poor texture and complex illumination.
• Hyperspectral imaging (HSI) offers potential for improved keypoint detection and matching in surgical environments.

Purpose of the Study:

• To investigate the efficacy of snapshot hyperspectral imaging (HSI) for keypoint detection and matching in surgical scenes.
• To address the limitations of RGB-based pipelines in texture-poor and complex illumination conditions encountered during MIS.
• To develop and evaluate a novel HSI-based model for enhanced 3D reconstruction in MIS.

Main Methods:

• Developed HyKey, a hybrid 3D-2D convolutional neural network for joint spatial-spectral feature extraction from HSI data.
• Trained the model using synthetic homographic augmentation and epipolar geometry constraints on a robotically acquired dual-camera RGB-HSI laparoscopic dataset.
• Benchmarked HyKey against established RGB-based methods like SuperPoint and ALIKE.

Main Results:

• The HSI-based HyKey model significantly outperformed RGB baselines on registered RGB frames.
• Achieved 96.62% mean matching accuracy and 67.18% mean average accuracy at 10 degrees on pose estimation.
• Demonstrated consistent improvements across multiple evaluation metrics, highlighting the benefits of spectral information.

Conclusions:

• Integrating spectral information from HSI cubes provides a robust approach for monocular 3D reconstruction in MIS.
• The developed HyKey model effectively addresses limitations in texture-poor surgical environments via enhanced spectral-spatial feature discrimination.
• The HyKey model and dataset are publicly available to facilitate further research in hyperspectral surgical imaging.