RGB-D based multi-modal deep learning for spacecraft and debris recognition.

Area of Science:

• Space situational awareness (SSA)
• Deep learning for computer vision
• Object recognition in space environments

Background:

• Accurate recognition of space objects is crucial for SSA tasks like satellite formation, servicing, and debris removal.
• Challenges in space imagery include diverse sensing conditions, noise, varied orbital scenarios, high contrast, low signal-to-noise ratio, and object size variations.
• Existing methods struggle with the complexity and variability of space imaging conditions.

Purpose of the Study:

• To propose a robust multi-modal deep learning solution for accurate recognition of space objects (spacecraft and debris).
• To address the challenges posed by diverse and difficult sensing conditions in actual space imagery.
• To evaluate the proposed solution using a novel, realistic space simulation dataset.

Main Methods:

• Feature extraction from RGB images using Convolutional Neural Network (CNN) models (ResNet, EfficientNet, DenseNet) and a Vision Transformer.
• Classification of depth images using an End-to-End CNN.
• A multi-modal approach combining decisions from RGB and depth-based models.
• Experiments conducted on the novel SPARK dataset (150k RGB, 150k depth images across 11 categories).

Main Results:

• The combined RGB-based Vision Transformer and Depth-based End-to-End CNN achieved high performance.
• Achieved accuracy of 85%, precision of 86%, recall of 85%, and an F1 score of 84%.
• Demonstrated superior performance compared to single-modality approaches on the SPARK dataset.

Conclusions:

• The proposed multi-modal deep learning solution is a feasible and effective approach for space object recognition.
• This method significantly improves capabilities for real-world Space Situational Awareness (SSA) applications.
• The SPARK dataset provides a valuable resource for advancing research in space object recognition.