Representations of facial features and surface quality in monkey area TE compared to neural network models

The shape and spacing of facial features are essential for the perception of emotional expressions and the identification of individuals. The surface quality of faces, such as skin texture, eye twinkle, and hair gloss, is valuable for estimating health status. A previous fMRI study in monkeys revealed that regions selective for faces overlapped regions selective for object gloss in the central inferior temporal (IT) cortex, suggesting that the surface quality of faces is processed in area TE. However, the representation of facial surface quality has not been directly examined in this area. To understand neural processing of surface quality in face images, neuronal activity was recorded in area TE of three monkeys while face images with different expressions, identities, and surface qualities were presented. The surface qualities included gloss modulations - high-gloss and low-gloss, and "style-transfer" where facial texture was replaced with fabric. These representations were compared to those in several models, including Convolutional Neural Networks and Vision Transformer. Our results revealed that the style-transferred face images strongly modulated activity in TE neurons and that some neurons were tuned to monkey facial expressions, whereas others were more influenced by surface quality. In contrast, the models showed relatively large contributions of surface-quality representation, particularly to the separation between the original and style-transferred faces, compared with those of facial expression and identity. This trend was observed even in models trained on the stylized ImageNet dataset, which was designed to reduce reliance on texture-based classification. These results suggest a strong influence of image style transfer on both TE neurons and models, the diversity of the facial representations by TE neurons, and the tendency of models to emphasize the style-transferred images regardless of training. These findings highlight a potential limitation in current artificial vision models and underscore the value of examining biological representations to inspire more robust and adaptable visual recognition in future model designs.