Talking to the Brain: Using Large Language Models as Proxies to Model Brain Semantic Features

Utilizing naturalistic stimuli while maintaining experimental rigor presents a significant challenge in neuroimaging research, primarily due to the complexities of annotation and high-dimensional analysis. To address this, we introduce a novel paradigm that leverages multimodal large language models (LLMs) as high-throughput, automated annotation tools to extract human-understandable semantic features. This "talking to the brain" approach employs a Visual Question Answering (VQA) strategy to transform complex visual scenes into structured semantic feature vectors, which are then used to predict voxel-wise brain activity. As a critical methodological validation, our model successfully replicated established neural correlates for categories such as face and building. Expanding beyond these benchmarks, we mapped the cortical activation patterns of 80 diverse semantic labels and constructed a data-driven brain semantic similarity space. This space revealed robust clusters reflecting both functional categories and real-world contextual associations. This innovative methodology offers a scalable solution for investigating brain semantic organization, overcoming the limitations of manual annotation and paving the way for more ecologically valid explorations of human cognition.