Augmented reality visualization of hyperspectral imaging classifications for image-guided brain tumor phantom...

Area of Science:

• Medical imaging
• Augmented reality
• Optical engineering

Background:

• Wearable augmented reality (AR) offers significant potential in medicine, particularly for image-guided surgery.
• Real-time tissue segmentation and classification are crucial for effective AR-based surgical guidance.
• Hyperspectral imaging (HSI) provides rapid, non-contact spectral data acquisition for tissue analysis.

Purpose of the Study:

• To develop and demonstrate a fast hyperspectral imaging and augmented reality (AR) system for real-time visualization of brain tissue.
• To integrate HSI data with machine learning for accurate tissue classification.
• To enable interactive, real-time manipulation of segmented tissue data within an AR environment for surgical applications.

Main Methods:

• Construction of a brain tissue phantom simulating blood vessels, tumor, and normal brain tissue.
• Application of a segmentation algorithm to hundreds of hyperspectral images for pixel-wise classification.
• Integration of HSI channels and segmentation labels with a HoloLens AR headset for superimposed visualization.
• Implementation of hand and voice controls for user interaction with the AR display.

Main Results:

• Successful classification of different tissue types within the phantom using HSI and segmentation.
• Feasible real-time superimposition of HSI data and segmentation labels onto the surgical field via AR.
• Intuitive user control over visualization elements (repositioning, rotation, visibility) through hand and voice commands.
• Demonstration of a practical and rapid HSI-AR technique.

Conclusions:

• The developed HSI-AR system shows feasibility for real-time image-guided brain surgery.
• This technology offers a convenient and multifaceted visualization tool for surgeons.
• The combination of HSI and AR presents a promising approach for enhancing surgical precision and safety.