Alliance: All-in-One Spectral-Spatial-Frequency Awareness Foundation Model.

Area of Science:

• Remote Sensing
• Computer Vision
• Signal Processing

Background:

• Foundation models (FMs) in remote sensing excel with spatial and spectral data but underutilize frequency domain information.
• Existing FMs struggle to bridge the semantic gap between frequency characteristics and original image content, hindering downstream task performance.

Purpose of the Study:

• To develop a novel foundation model, Alliance, that comprehensively integrates spectral, spatial, and frequency information for enhanced remote sensing analysis.
• To address the limitations of current FMs in capturing hidden features within the frequency domain and maintaining multi-domain coherence.

Main Methods:

• Proposed the All-in-One Spectral-Spatial-Frequency Awareness Foundation Model (Alliance) framework.
• Introduced a progressive frequency decoding mechanism mimicking human visual cognition to minimize multi-domain information gaps.
• Developed a triple-domain fusion attention module for integrated processing of amplitude, phase, and spectral-spatial relationships.
• Implemented frequency embedding with specialized token initialization for fine-grained frequency band modeling.

Main Results:

• Alliance demonstrated superior performance across six downstream remote sensing tasks.
• The progressive frequency decoding effectively extracts subtle image patterns difficult to observe in raw pixel values.
• The triple-domain fusion attention and frequency embedding enabled comprehensive feature integration and fine-grained modeling.

Conclusions:

• Alliance represents a significant advancement in foundation models for remote sensing by effectively leveraging the frequency domain.
• The proposed methods successfully bridge the semantic gap between different data domains, leading to improved generalizability and performance.
• The Yellow River dataset provides a robust benchmark for evaluating FM capabilities in challenging cross-domain remote sensing scenarios.