Masked image modeling in medical hyperspectral imaging: reconstruction evaluation and downstream tasks

Area of Science:

• Medical Imaging
• Deep Learning
• Computer Vision

Background:

• Self-supervised pre-training boosts deep learning performance in NLP and computer vision.
• Limited research exists on pre-training for medical imaging, potentially hindering downstream task performance.
• Hyperspectral imaging (HSI) captures rich spectral and spatial information crucial for tissue analysis.

Purpose of the Study:

• To develop and evaluate a masked autoencoding framework for pre-training hyperspectral images (HSI).
• To assess the effectiveness of this pre-trained model for ex vivo tissue classification.
• To improve the encoding of spatial and spectral features in medical imaging applications.

Main Methods:

• Utilized a state-of-the-art pre-training architecture with a masked image modeling scheme on an internal HSI dataset.
• Implemented a network with sequential spectral and spatial attention for efficient feature encoding.
• Evaluated pre-training using reconstruction visualization and Mean Absolute Error (MAE), and finetuned for abdominal tissue classification on an unseen validation dataset.

Main Results:

• Achieved a Mean Absolute Error (MAE) of 0.0294 during pre-training on the validation dataset.
• The finetuned network achieved 87.9% accuracy in classifying 17 classes of abdominal tissues with frozen weights.
• Demonstrated effective encoding of spatial and spectral features from ex vivo tissues.

Conclusions:

• The proposed masked autoencoding framework effectively pre-trains hyperspectral images for medical applications.
• This approach shows significant potential for improving downstream tasks like tissue classification and segmentation.
• Highlights the importance of thorough evaluation of pre-training strategies in medical imaging.