Integrating Vision and Olfaction via Multi-Modal LLM for Robotic Odor Source Localization.

Area of Science:

• Robotics
• Artificial Intelligence
• Sensor Fusion

Background:

• Odor source localization (OSL) is crucial for autonomous agents in unknown environments.
• Traditional OSL algorithms often rely on single sensory modalities or complex supervised learning.
• Environmental complexities like non-unidirectional airflow can disrupt olfaction-only systems.

Purpose of the Study:

• To develop and validate a novel OSL algorithm for mobile robots.
• To integrate vision and olfaction sensor modalities using large language models (LLMs).
• To improve OSL performance in challenging real-world conditions.

Main Methods:

• Developed an LLM-based OSL algorithm with 'High-level Reasoning' and 'Low-level Action' modules.
• Integrated multi-modal sensor data (vision and olfaction) into LLM prompts.
• Implemented and tested the algorithm on a mobile robot in complex, real-world environments.

Main Results:

• The proposed LLM-based algorithm demonstrated superior performance compared to 'olfaction-only', 'vision-only', and supervised learning fusion algorithms.
• Achieved higher success rates and reduced average search times in both unidirectional and non-unidirectional airflow scenarios.
• Effectively navigated complex environments with obstacles and disrupted airflow.

Conclusions:

• LLM-based multi-modal sensor fusion offers a robust and efficient approach to odor source localization.
• The zero-shot reasoning capabilities of LLMs eliminate the need for manual knowledge encoding or custom supervised models.
• This technology significantly advances autonomous agent capabilities in practical search and navigation tasks.