Cognitively-Inspired Tokens Overcome Egocentric Bias in Multimodal Models.

Area of Science:

• Artificial Intelligence
• Cognitive Science
• Computer Vision

Background:

• Multimodal language models (MLMs) excel at semantic tasks but struggle with spatial reasoning, particularly perspective-taking.
• Current MLMs exhibit egocentric bias, limiting their ability to perform allocentric reasoning.
• Human spatial cognition offers insights into developing more advanced AI reasoning capabilities.

Purpose of the Study:

• To introduce a novel method for enhancing allocentric reasoning in MLMs.
• To improve visual perspective-taking abilities in AI models.
• To investigate the potential for MLMs to support human-like spatial reasoning.

Main Methods:

• Developed "perspective tokens," specialized embeddings encoding orientation via embodied or abstract representations.
• Integrated these tokens into the LLaVA-1.5-13B model.
• Evaluated performance on synthetic and naturalistic benchmarks, including Isle Bricks V2, COCO, and 3DSRBench.

Main Results:

• The integration of perspective tokens significantly improved performance on level-2 visual perspective-taking tasks.
• Rotation-based tokens demonstrated generalization capabilities to non-human reference agents.
• Representational analyses indicated that fine-tuning enhances latent orientation sensitivity in MLMs.

Conclusions:

• Embedding cognitively grounded spatial structure into token space is an effective, lightweight, and model-agnostic approach for perspective-taking.
• MLMs possess precursors for allocentric reasoning, but require appropriate internal structure for effective utilization.
• This method facilitates more human-like spatial reasoning in AI systems.