Unraveling the geometry of visual relational reasoning

Humans readily generalize abstract relations, such as recognizing "constant" in shape or color, whereas neural networks struggle to do so, limiting their flexible reasoning. We ask what properties of internal representations enable abstract relational generalization and propose a geometric framework for understanding relational reasoning. We introduce SimplifiedRPM, a controlled benchmark that isolates abstract relational structure and enables rigorous evaluation of generalization to unseen rules. We collect human behavioral data on the task to quantify relational difficulty and enable direct comparison between models and human reasoning. Testing four models-ResNet-50, Vision Transformer, Wild Relation Network, and Scattering Compositional Learner (SCL)-we find that SCL generalizes best and most closely aligns with human difficulty ordering. Using a geometric approach, we model relational rules as manifolds in representation space and show that interpretable geometric quantities accurately predict generalization performance. Layer-wise analysis reveals distinct geometric strategies adopted by different architectures. We further uncover a trade-off between representation signal and dimensionality, with learned and unseen relations aligning with a common low-dimensional subspace. Finally, we show that directly optimizing the representation geometry improves relational generalization in this controlled benchmark, demonstrating that it can be leveraged to guide training. Together, our results demonstrate that representation geometry provides a principled framework for relational generalization in SimplifiedRPM and opens promising avenues for extending geometric analysis to broader cognitive tasks.