A functional MRI study of a picture-sentence verification task: evidence of attention shift to the grammatical...

Mapping the meaning of a sentence onto visual entities is a fundamental process of daily language use, but it is unclear how attention in the visual context influences sentence comprehension. Aiming to examine this problem, we conducted a picture-sentence matching experiment with scanning using functional MRI. In the experiment, a moving picture describing an event with two colored objects was presented on a screen. A visual cue was flashed at the position of an object's appearance just before the event presentation, and participants were instructed to pay attention to the visually cued object in the picture. They were then required to read a simple Japanese sentence and to verify whether it correctly described the previous event. To examine the effects of visual cueing, we defined two conditions on the basis of the relationship between the visually cued object in an event and the grammatical subject of the subsequent sentence. When comparing the conditions in which the visually cued object was incongruent with the grammatical subject to the congruent conditions, participants showed a lower hit rate, and the right frontal eye field, which is known to be the region related to attention shift, was more activated. These findings suggest that the attention was initially allocated to an object encoded as the grammatical subject in the process of linking the content of a sentence with a visual event. Therefore, the attention was shifted from the cued object to the other object under the conditions discussed above.