Event Knowledge in Large Language Models: The Gap Between the Impossible and the Unlikely.

Area of Science:

• Computational Linguistics
• Cognitive Science
• Artificial Intelligence

Background:

• Word co-occurrence patterns in language corpora encode conceptual knowledge.
• Large language models (LLMs) leverage these patterns for semantic tasks and world knowledge acquisition.
• The extent of LLMs' generalized knowledge of common events is understudied.

Purpose of the Study:

• To investigate whether pretrained LLMs can differentiate plausible from implausible descriptions of agent-patient interactions.
• To assess the extent of event knowledge acquired by LLMs trained on distributional linguistic patterns.
• To compare LLM performance with other distributional language models in understanding event semantics.

Main Methods:

• Tested five pretrained LLMs (BERT to MPT) on curated sets of minimal sentence pairs (n=1215).
• Compared LLM likelihood assignments for plausible vs. implausible event descriptions.
• Analyzed factors influencing LLM scores, including surface-level features and syntactic/semantic variants.

Main Results:

• LLMs consistently assigned higher likelihood to possible events over impossible ones (e.g., 'teacher bought laptop' vs. 'laptop bought teacher').
• LLMs showed less consistent preferences for likely over unlikely events (e.g., 'nanny tutored boy' vs. 'boy tutored nanny').
• LLM performance was influenced by sentence plausibility and surface features, generalizing better across syntax than semantics.

Conclusions:

• Pretrained LLMs acquire substantial event knowledge from distributional linguistic patterns, particularly distinguishing possible from impossible events.
• A notable gap exists in LLMs' ability to consistently differentiate likely from unlikely events.
• Sentence plausibility is an organizing dimension within LLM internal representations, but nuanced event understanding requires further development.