Microscopic Hyperspectral Image Classification Based on Fusion Transformer With Parallel CNN.

Area of Science:

• Medical Imaging
• Computer Vision
• Artificial Intelligence

Background:

• Microscopic hyperspectral image (MHSI) analysis is crucial in medicine.
• Convolutional Neural Networks (CNNs) struggle with long-range spectral dependencies in high-dimensional MHSI.
• Transformers excel at long-range dependencies but lack detailed spatial feature extraction.

Purpose of the Study:

• To propose a novel classification framework, Fusion Transformer (FUST), for MHSI.
• To leverage the strengths of both Transformer and CNN architectures for improved MHSI classification.
• To enhance the extraction of spectral and spatial features in MHSI data.

Main Methods:

• A parallel framework integrating Transformer and CNN branches was developed.
• The Transformer branch captures long-range spectral dependencies.
• The CNN branch extracts multiscale spatial features, followed by a feature fusion module.

Main Results:

• The Fusion Transformer (FUST) framework demonstrated superior performance on three MHSI datasets.
• FUST outperformed existing state-of-the-art methods in MHSI classification tasks.
• The integrated approach effectively fused spectral and spatial features for enhanced identification.

Conclusions:

• The proposed FUST framework offers a powerful new approach for MHSI classification.
• Combining Transformer and CNN architectures addresses limitations of individual models.
• FUST shows significant potential for advancing medical image analysis using hyperspectral data.