Visual perception of materials and their properties

Misidentifying materials-such as mistaking soap for pâté, or vice versa-could lead to some pretty messy mishaps. Fortunately, we rarely suffer such indignities, thanks largely to our outstanding ability to recognize materials-and identify their properties-by sight. In everyday life, we encounter an enormous variety of materials, which we usually distinguish effortlessly and without error. However, despite its subjective ease, material perception poses the visual system with some unique and significant challenges, because a given material can take on many different appearances depending on the lighting, viewpoint and shape. Here, I use observations from recent research on material perception to outline a general theory of material perception, in which I suggest that the visual system does not actually estimate physical parameters of materials and objects. Instead-I argue-the brain is remarkably adept at building 'statistical generative models' that capture the natural degrees of variation in appearance between samples. For example, when determining perceived glossiness, the brain does not estimate parameters of the BRDF. Instead, it uses a constellation of low- and mid-level image measurements to characterize the extent to which the surface manifests specular reflections. I argue that these 'statistical appearance models' are both more expressive and easier to compute than physical parameters, and therefore represent a powerful middle way between a 'bag of tricks' and 'inverse optics'.