NavBLIP: a visual-language model for enhancing unmanned aerial vehicles navigation and object detection

Area of Science:

• Robotics and Artificial Intelligence
• Computer Vision
• Machine Learning

Background:

• Traditional Unmanned Aerial Vehicle (UAV) navigation and object detection methods struggle with dynamic environments, lacking robustness and real-time efficiency.
• Unimodal deep learning and handcrafted features limit adaptability and generalization in complex scenarios.
• Effective integration of multimodal data is crucial for advanced UAV capabilities.

Purpose of the Study:

• To introduce NavBLIP, a novel visual-language model for enhanced UAV navigation and object detection.
• To leverage multimodal data for improved robustness and computational efficiency in UAVs.
• To address the limitations of existing methods in dynamic and complex environments.

Main Methods:

• Developed NavBLIP, a visual-language model utilizing transfer learning and a Nuisance-Invariant Multimodal Feature Extraction (NIMFE) module.
• The NIMFE module disentangles relevant features from complex visual and environmental inputs.
• Implemented a multimodal control strategy for dynamic, context-specific feature selection to optimize real-time performance.

Main Results:

• NavBLIP demonstrated superior performance over state-of-the-art models in accuracy, recall, and computational efficiency on benchmark datasets (RefCOCO, CC12M, OpenImages).
• Ablation studies confirmed the significant contribution of NIMFE and transfer learning to performance gains.
• The model shows high potential for real-time UAV applications requiring adaptability.

Conclusions:

• NavBLIP effectively enhances UAV navigation and object detection through multimodal data integration.
• The NIMFE module and transfer learning are critical components for improved performance and adaptability.
• NavBLIP offers a promising solution for real-time, efficient, and robust UAV operations in diverse environments.