Situating vision in the world

Recently, there has been a great deal of interest in what has been called 'situated cognition', which has included claims that certain forms of representation are inadequate for modeling active organisms or agents such as humans and robots. In this article, I suggest that a weakness in classical theories of visual representation is the way in which representations connect with the real world, which may account for many of the concerns expressed by the situated cognition community. Specifically, I claim that what current theories lack is any provision for a certain form of direct, preconceptual connection between objects in the visual world (visual objects or proto-objects) and their representations in the visual system. This type of connection is akin to what philosophers and semanticists have referred to as an 'indexical' or 'demonstrative' reference and what some cognitive scientists have referred to as 'deictic pointers'. I explain why such a mechanism is needed and suggest that many workers have, in fact, been studying precisely this under the term 'visual index'. The visual index hypothesis is illustrated with the results of some relevant experiments, including multiple object tracking, visual routines and subset-selected visual searches. Indexing theory provides a synthesis that has profound implications for explaining a wide range of psychophysical findings, certain results in infant cognitive development and also some ancient problems in the philosophy of mind.