Multi-View Images Suffice 3D Reasoning Through Chain-of-Thought Selection and Question-Guided Fusion.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Robotics

Background:

• 3D reasoning is essential for robotics and autonomous driving.
• Acquiring 3D data is costly, prompting research into using multi-view images for 3D reasoning with Large Language Models (LLMs).
• Existing multi-view methods suffer from redundant views and loss of fine-grained correlations.

Purpose of the Study:

• To develop a 3D reasoning method that overcomes the limitations of existing multi-view approaches.
• To enable LLMs to perform 3D reasoning using only multi-view images, without 3D-specific data.
• To improve the efficiency and accuracy of 3D scene understanding.

Main Methods:

• Proposed 3DMulti-LLM, comprising a Chain-of-Thought (COT) selector, a question-guided fusion block, and pre-trained LLMs.
• The COT selector identifies question-relevant multi-view images, reducing interference from irrelevant viewpoints.
• A question-guided fusion block integrates multi-view features through inter-viewpoint interaction, enabling direct 3D scene reasoning via multi-view images.

Main Results:

• 3DMulti-LLM achieves state-of-the-art performance on 3D reasoning tasks.
• Demonstrated significant improvements over existing 3D-input-free methods, surpassing them by +12.2% on ScanQA and +7.1% on 3DMV-VQA.
• The method successfully reasons about 3D scenes using only multi-view images, eliminating the need for point cloud data.

Conclusions:

• 3DMulti-LLM offers an effective and efficient solution for 3D reasoning using multi-view images.
• The proposed approach successfully transfers LLMs' 2D reasoning capabilities to 3D environments without costly 3D data.
• This work paves the way for more accessible and powerful 3D understanding in AI applications.