Brain-inspired large model mindreading

Multimodal large language models (MLLMs) demonstrate significant limitations in visual Theory of Mind (ToM) abilities compared to humans. Investigating the neural mechanisms underlying human mind-reading not only addresses critical gaps in visual ToM research but also provides valuable insights for MLLM optimization. Using fMRI data from 83 participants, we systematically compared neural processing patterns between two conditions in visual ToM tasks: MLLM-incorrect/human-correct (MLLMI) and mutually correct (MLLMC). The questionnaire results indicated that, compared with the MLLMC condition, human confidence was significantly lower under the MLLMI condition, and expectations for MLLM performance were also lower. Natural language analysis revealed that, relative to MLLM responses, human responses were more closely aligned with the question context, more concise, and exhibited greater certainty. Neuroimaging results indicated significantly stronger activation in bilateral precuneus and middle temporal gyrus in MLLMI. Furthermore, we observed enhanced functional connectivity in networks associated with task coordination and attention allocation. Leveraging these neural signatures, we constructed multiple prediction models for decoding the two conditions (MLLMI vs. MLLMC), among which the 2-layer Transformer model achieved the highest classification accuracy of 78.6%. Extending these findings, we propose the Knowledge-Thinking-Adaptation (KTA) framework, which integrates memory retrieval, divergent thinking, and multi-level attention mechanisms to provide a potential roadmap for future work in developing AI systems with human-like visual ToM capabilities.