Language with vision: A study on grounded word and sentence embeddings.

Area of Science:

• Cognitive Science
• Natural Language Processing
• Computer Vision

Background:

• Grounding language in vision aims to create cognitively plausible representations by integrating visual perception with text.
• Current challenges include optimizing the balance between textual and embodied representations and determining the scope of visual grounding's effectiveness for abstract versus concrete words.
• The extent to which visual knowledge improves embedding quality remains an active research question.

Purpose of the Study:

• To investigate the effectiveness of visual grounding for both abstract and concrete words.
• To propose an optimal method for bridging the gap between text and vision in language representations.
• To determine the advantage of perceptual image knowledge for acquiring high-quality word embeddings.

Main Methods:

• Developed a computational grounding model to align pre-trained word embeddings with visual information.
• The model preserves textual distributional statistics while incorporating visual grounding.
• Employed a learned alignment strategy to indirectly ground unseen and abstract words.

Main Results:

• Visual grounding demonstrated benefits for both concrete and abstract words, supporting the indirect theory of abstract concepts.
• The proposed model effectively balances language and vision, improving word embeddings.
• Contextualized embeddings, like BERT, showed advantages when trained on modest corpus sizes.

Conclusions:

• Visual grounding is a valuable technique applicable to a wide range of words, including abstract concepts.
• The developed model offers an effective approach to integrating visual information into language representations.
• The findings have implications for improving language models and understanding cognitive processes in language acquisition.