Logic, inference, understanding: cross-domain generalization for generative language models.

Area of Science:

• Artificial Intelligence
• Computational Linguistics
• Formal Semantics

Background:

• Neural Natural Language Inference (NLI) models show high performance but struggle with generalization.
• NLI is often used as a proxy for Natural Language Understanding (NLU), based on inferentialist semantics.
• Existing research questions the generalization capabilities of NLI models beyond their training data.

Purpose of the Study:

• To disentangle the problem of generalization in NLI by introducing distinctions between different notions of generalization.
• To analyze in-domain vs. cross-domain and linguistic vs. inferential generalization.
• To investigate the inferential generalization power of autoregressive NLI models.

Main Methods:

• Drawing on formal logic and semantics to define distinct types of generalization.
• Designing experiments to test the inferential generalization capabilities of NLI models.
• Leveraging theoretical contributions to analyze model performance.

Main Results:

• Distinguishing between generalization types clarifies model limitations.
• Experiments reveal specific weaknesses in the inferential generalization of autoregressive NLI models.
• Theoretical insights provide a framework for evaluating NLI model generalization.

Conclusions:

• The ability of NLI models to generalize, particularly inferentially, remains a significant challenge for NLU.
• A clearer theoretical framework is needed to accurately assess and improve NLI model generalization.
• Future research should focus on enhancing the inferential reasoning capabilities of NLI models.