Response dynamics in macaque ventral stream recapitulate the visual hierarchy

Neuronal responses in macaque inferotemporal cortex (IT) are typically summed over broad time windows, obscuring how selectivity evolves during the response. Using chronic, multi-site recordings in 5 male monkeys and large image sets, we observed a pronounced temporal cascade of response properties in IT: in different time bins distinct images evoked the strongest responses. Neuronal image-feature tuning shifted over the response duration: from large objects to small, from objects with few parts to many, from objects with simple perimeters to more complex perimeters, and from low spatial frequency to high. Representation similarity between IT neurons and deep neural networks (DNN) peaked earlier for shallow DNN layers than for deeper layers, suggesting a feature hierarchy unfolding over time in IT. These same image-feature metrics also quantitatively characterized DNN hierarchies from shallow to deep layers.Significance statement: This study proposes a novel interpretation of response dynamics in inferotemporal (IT) cortex. A shift from simple to more complex images was observed in face-selective sites in IT. Further shallow layers of DNNs correlated better with early parts of IT responses, and deeper layers with later parts of the response. Low-level image features peaked early in the response and showed stronger correlations with shallow DNN layers, providing a quantitative measure of changing complexity along both the visual hierarchy and DNN hierarchies.