Sea surface suspended concentrations prediction based on pre-trained transformer.

Area of Science:

• Marine Engineering
• Remote Sensing
• Data Science

Background:

• Monitoring sea surface suspended sediments (SSSC) is crucial for marine engineering, habitat ecology, and environmental studies.
• Current methods using suit or satellite data face challenges in spatial sampling and generalization.
• Limited data availability hinders the performance of traditional deep learning models for SSSC tasks.

Purpose of the Study:

• To develop an advanced deep learning model for accurate SSSC reconstruction.
• To address the limitations of existing SSSC monitoring techniques.
• To explore the effectiveness of leveraging pre-trained models from natural language processing for SSSC tasks.

Main Methods:

• Proposed a pre-trained transformer model adapted from natural language processing (NLP) for SSSC reconstruction.
• Designed input embedding layers to integrate remote sensing data with the pre-trained model.
• Utilized knowledge transfer by freezing specific layers during fine-tuning for the SSSC task.

Main Results:

• The proposed pre-trained transformer model achieved state-of-the-art performance in SSSC reconstruction.
• Demonstrated the efficacy of transferring knowledge from large-scale NLP models to specialized tasks.
• Indicated the potential of large pre-trained models to improve performance on data-scarce scientific domains.

Conclusions:

• Pre-trained transformer models offer a powerful solution for challenging SSSC monitoring tasks.
• Knowledge transfer from massive datasets can significantly enhance performance in specialized scientific applications.
• This approach paves the way for more accurate and reliable environmental monitoring in marine engineering.