SpatialPrompting: pose-aware keyframe prompting for 3D spatial QA toward smart indoor environments

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Multimodal large language models (MLLMs) show promise for complex reasoning tasks.
• Existing methods often require task-specific fine-tuning and 3D-specific data.
• Efficiently leveraging MLLMs for 3D spatial reasoning remains a challenge.

Purpose of the Study:

• To introduce SpatialPrompting, a practical, training-free, and model-agnostic framework for 3D spatial reasoning.
• To enable viewpoint-aware reasoning within a single structured prompt using a pose-aware, keyframe-driven strategy.
• To evaluate the framework's performance and generalizability across different MLLMs.

Main Methods:

• Developed a pose-aware, keyframe-driven prompting strategy.
• Selected diverse keyframes using vision-language similarity, Mahalanobis distance, field of view, and image sharpness.
• Verbalized camera poses to enable multi-view reasoning within a structured prompt.

Main Results:

• Achieved competitive performance on the ScanQA dataset.
• Showed strong results on the Complex Spatial QA (CSQA) dataset, improving accuracy by +16 over a baseline.
• Demonstrated consistent performance across proprietary (GPT-4o, Gemini) and open-source (Qwen) models.

Conclusions:

• SpatialPrompting provides a scalable and effective approach for 3D spatial reasoning with MLLMs.
• The framework eliminates the need for task-specific training, shifting costs to flexible inference.
• Generalizes across various MLLMs, offering a reusable solution for evolving AI models.