Representations of smells: The next frontier for language models?

Area of Science:

• Cognitive Science
• Natural Language Processing
• Computational Linguistics

Background:

• Human cognition develops through environmental interaction, unlike disembodied language models (LMs).
• A gap exists in understanding LMs' ability to extract sensory information from text, particularly for underrepresented senses like olfaction.
• Olfaction presents a unique challenge for linguistic and cognitive modeling due to its limited representation in natural language.

Purpose of the Study:

• To evaluate the capacity of LMs to recover olfactory information from natural language.
• To investigate the conditions under which LMs can approximate human odor perception.
• To explore the relationship between symbolic representations and perceptual experience using LMs.

Main Methods:

• Systematic evaluation of three generations of LMs: static word embeddings (Word2Vec, FastText), encoder-based (BERT), and decoder-based large LMs (LLMs; GPT-4o, Llama 3.1).
• Testing under nearly 200 training configurations to assess proficiency in acquiring olfactory information from text.
• Utilizing three experimental odor datasets: odor similarity ratings, imagined odor similarities from word labels, and odor-to-label ratings as benchmarks.

Main Results:

• LMs demonstrate the ability to accurately represent olfactory information under specific training conditions.
• Static, simpler models performed best in capturing odor-perceptual similarities in certain configurations.
• GPT-4o excelled in simulating olfactory-semantic relationships, particularly on word-based odor similarity assessments.

Conclusions:

• Natural language encodes latent olfactory information retrievable by LMs to varying degrees.
• The findings suggest LMs can be valuable tools for studying the link between symbolic representation and perceptual experience.
• Different LM architectures and training configurations yield varying successes in olfactory information recovery.