Grounding spatial relations in text-only language models.

Area of Science:

• Natural Language Processing
• Computer Vision
• Artificial Intelligence

Background:

• Spatial reasoning is a complex cognitive task crucial for understanding visual scenes.
• Current models often rely on multimodal inputs (vision and language) for spatial understanding.
• Text-only models have limitations in grounding spatial relations without explicit location cues.

Purpose of the Study:

• To investigate if text-only Language Models (LMs) can learn to ground spatial relations using explicit object location information.
• To evaluate the impact of pretraining on a synthetic dataset for improving spatial reasoning in LMs.
• To compare the performance of text-only LMs with location grounding against existing Vision-and-Language Models.

Main Methods:

• Verbalizing images from the Visual Spatial Reasoning (VSR) dataset with object location tokens derived from an object detector.
• Pretraining a text-only LM on a synthetically generated dataset incorporating spatial relations and location data.
• Evaluating the LM's ability to distinguish real from fake spatial relations on the VSR dataset.

Main Results:

• Text-only LMs, when provided with location tokens and pretraining, significantly improved performance on spatial relation tasks.
• The proposed method achieved state-of-the-art results on the VSR dataset, outperforming Vision-and-Language Models.
• LMs demonstrated generalization capabilities, learning beyond explicitly encoded spatial rules in the synthetic data.

Conclusions:

• Explicit object location information enables text-only LMs to effectively ground spatial relations.
• Pretraining on relevant synthetic data is crucial for achieving high performance in text-only spatial reasoning.
• This research advances the capabilities of text-only models in complex reasoning tasks, challenging the necessity of multimodal inputs for certain spatial understanding.