Towards robust remote sensing visual question answering with spectral expert adaptation and group-relative...

Area of Science:

• Artificial Intelligence
• Computer Vision
• Remote Sensing

Background:

• Transformer-based vision-language large language models (V-L LLMs) show promise in multimodal understanding but face challenges in remote sensing (RS) visual question answering (VQA) due to domain shift and high adaptation costs.
• Existing V-L LLMs struggle with the spatially organized nature of remote sensing scenes and require significant resources for fine-tuning.

Purpose of the Study:

• To introduce SpectralLoRA-R1, an efficient method for adapting V-L LLMs to the remote sensing visual question answering (RS-VQA) domain.
• To address the limitations of domain shift and high adaptation costs in applying V-L LLMs to RS-VQA tasks.
• To improve the performance of RS-VQA models without requiring full model fine-tuning.

Main Methods:

• Developed Mixture-of-Experts SpectralLoRA (MoE-SpectralLoRA) to adapt frozen pretrained weights using low-rank residuals in a fixed spectral basis.
• Implemented a lightweight MoE router to dynamically combine SpectralLoRA experts for different image-question pairs.
• Applied Generalized Reinforcement Policy Optimization (GRPO) with verifiable rewards for post-training to refine answer format, consistency, and correctness.

Main Results:

• SpectralLoRA-R1 demonstrated improved performance across multiple RS-VQA benchmarks compared to previous methods.
• The method successfully adapted V-L LLMs to the RS domain while keeping the backbone model frozen.
• Achieved better RS-VQA accuracy by combining fixed-basis spectral adaptation with verifier-guided post-training.

Conclusions:

• Fixed-basis spectral adaptation is an effective strategy for domain adaptation in RS-VQA.
• Verifier-guided post-training can enhance V-L LLM performance in remote sensing without full model fine-tuning.
• SpectralLoRA-R1 offers a cost-effective and efficient approach to advance remote sensing visual question answering capabilities.