Visual Heuristics for Verb Production: Testing a Deep-Learning Model With Experiments in Japanese

Area of Science:

• Cognitive Science
• Linguistics
• Computer Science

Background:

• Verb tense and aspect morphology are often linked to event features such as telicity.
• The cognitive mechanisms by which speakers identify these event features from visual scenes remain largely unknown.

Purpose of the Study:

• To investigate how speakers identify event features from visual scenes to select verb morphology.
• To develop and test a computational model of verb production from visual input.

Main Methods:

• Japanese speakers described computer-generated animations varying in visual features related to telicity.
• A deep-learning model was created to simulate verb production from visual input.
• Model predictions regarding video duration, verb complexity, and endpoint input were tested.

Main Results:

• Adults and children successfully used goal information in animations to select appropriate past and progressive verb forms.
• A deep-learning model replicated human-like verb form distributions and accurately used visual cues for morphology selection.
• Model predictions confirmed that past tense production increases with endpoint input and video duration relates to verb complexity.

Conclusions:

• Verb production is tightly linked to visual heuristics that facilitate event understanding.
• This research provides insights into the interplay between visual perception and grammatical morphology.
• The findings support a connectionist approach to modeling language production grounded in visual input.