Leveraging Context for Perceptual Prediction Using Word Embeddings

Word embeddings derived from large language corpora have been successfully used in cognitive science and artificial intelligence to represent linguistic meaning. However, there is continued debate as to how well they encode useful information about the perceptual qualities of concepts. This debate is critical to identifying the scope of embodiment in human semantics. If perceptual object properties can be inferred from word embeddings derived from language alone, this suggests that language provides a useful adjunct to direct perceptual experience for acquiring this kind of conceptual knowledge. Previous research has shown mixed performance when embeddings are used to predict perceptual qualities. Here, we tested if we could improve performance by leveraging the ability of Transformer-based language models to represent word meaning in context. To this end, we conducted two experiments. Our first experiment investigated noun representations. We generated decontextualized ("charcoal") and contextualized ("the brightness of charcoal") Word2Vec and BERT embeddings for a large set of concepts and compared their ability to predict human ratings of the concepts' brightness. We repeated this procedure to also probe for the shape of those concepts. In general, we found very good prediction performance for shape, and a more modest performance for brightness. The addition of context did not improve perceptual prediction performance. In Experiment 2, we investigated representations of adjective-noun phrases. Perceptual prediction performance was generally found to be good, with the nonadditive nature of adjective brightness reflected in the word embeddings. We also found that the addition of context had a limited impact on how well perceptual features could be predicted. We frame these results against current work on the interpretability of language models and debates surrounding embodiment in human conceptual processing.