SSF-Net: Spatial-Spectral Fusion Network With Spectral Angle Awareness for Hyperspectral Object Tracking

Area of Science:

• Computer Vision
• Remote Sensing
• Signal Processing

Background:

• Hyperspectral video (HSV) provides rich spatial, spectral, and temporal data, ideal for challenging object tracking scenarios.
• Existing HSV tracking methods often underutilize spectral information and struggle with feature representation.
• Current approaches frequently rely on RGB trackers, limiting the full potential of hyperspectral data.

Purpose of the Study:

• To propose a novel spatial-spectral fusion network with spectral angle awareness (SSF-Net) for improved hyperspectral (HS) object tracking.
• To enhance spectral feature extraction and fusion to achieve complementary object representations.
• To develop a method that leverages both HS and RGB modalities for robust tracking.

Main Methods:

• A spatial-spectral feature backbone ($S^2$FB) for joint texture and spectrum representation.
• A spectral attention fusion module (SAFM) to correlate HS and RGB modalities for robust feature fusion.
• A spectral angle awareness module (SAAM) and loss (SAAL) for precise object localization based on spectral similarity.
• A weighted prediction method combining HS and RGB motion predictions.

Main Results:

• The proposed SSF-Net demonstrates superior performance compared to state-of-the-art trackers on benchmark datasets (HOTC-2020, HOTC-2024, BihoT).
• The network effectively extracts and fuses spatial and spectral features, leading to more accurate object tracking.
• The spectral angle awareness mechanism significantly improves localization accuracy.

Conclusions:

• SSF-Net offers a significant advancement in hyperspectral video object tracking by effectively utilizing spectral information.
• The proposed fusion strategy and spectral awareness modules enhance tracking robustness and accuracy.
• The method provides a strong foundation for future research in HS object tracking.