Cross-modal invariants as the foundational basis of emerging brain circuitry representations

Our perception of input from one sensory modality is malleable by inputs from other sensory modalities, also when they are non-coincident in time. This function has been critical for survival throughout evolution. How it arises may reveal fundamental principles of brain operation and inform the design of more intuitive human-machine interfaces and Virtual Reality (VR). Here we outline how the brain can acquire what we refer to as cross-modal invariants, brain representations that are applicable across any context to integrate information from multiple modalities. We begin by reviewing evidence that early unimodal brain circuitry processing of tactile inputs in the CNS, before the cortex, is organized around kinematic invariants. Published data is compatible with kinematic invariants in retinal processing, too. We show how such invariants can form natural components of biologically adapted versions of the previously introduced plenhaptic and plenoptic functions, plus a proposed matching plenauditory function. Behavioral observations, we argue, indicate that cross-modal invariants in addition rely on the capability of recording forward information in working memory and therefore this function should be dominated by the cortex. Whereas the unimodal invariants appear to reside in pre-cortical structures and have a ground truth in physics, we explain why cross-modal invariants have not. Therefore cross-modal invariants will need to be individually learned. This likely renders them intelligible only from the point of view of the intrinsic functional organization of the circuitry operations that emerge from that learning. Hence, cross-modal invariants can be expected to be more individually variable and furthermore ubiquitous in the representation space of the cortical circuitry, which can explain why cross-modal integration in the brain has proven so elusive to map out and understand.