Template construction grammar: from visual scene description to language comprehension and agrammatism

Area of Science:

• Neuroscience
• Computational Linguistics
• Cognitive Science

Background:

• The brain integrates linguistic, perceptual, and world knowledge for communication.
• Schema theory provides a computational framework for perceptuo-motor coordination.
• Understanding language-vision interaction is key to communication.

Purpose of the Study:

• To model the integration of linguistic, perceptual, and world knowledge in communication.
• To apply schema theory to a computational model of language production and comprehension.
• To investigate language processing in agrammatic aphasia patients.

Main Methods:

• Developed the SemRep/TCG model for language production, linking semantic scene representation with Template Construction Grammar.
• Extended TCG to model language comprehension, distinguishing between heavy and light semantics.
• Analyzed comprehension performances of agrammatic aphasic patients using sentence-picture matching tasks.

Main Results:

• The SemRep/TCG model successfully generates verbal descriptions from visual scene representations.
• The extended TCG model offers a computational framework for analyzing language processes, including interactions between grammar, world knowledge, and visual information.
• The model provides insights into comprehension deficits in agrammatic aphasia.

Conclusions:

• The TCG model computationally links language production and comprehension with visual perception.
• The model quantitatively analyzes distributed language dynamics and interactions between different information types.
• This work advances neurolinguistic computational modeling and its challenges for neuroinformatics.