Brain-model neural similarity reveals abstractive summarization performance

Deep language models (DLMs) have exhibited remarkable language understanding and generation capabilities, prompting researchers to explore the similarities between their internal mechanisms and human language cognitive processing. This study investigated the representational similarity (RS) between the abstractive summarization (ABS) models and the human brain and its correlation to the performance of ABS tasks. Specifically, representational similarity analysis (RSA) was used to measure the similarity between the representational patterns (RPs) of the BART, PEGASUS, and T5 models' hidden layers and the human brain's language RPs under different spatiotemporal conditions. Layer-wise ablation manipulation, including attention ablation and noise addition was employed to examine the hidden layers' effect on model performance. The results demonstrate that as the depth of hidden layers increases, the models' text encoding becomes increasingly similar to the human brain's language RPs. Manipulating deeper layers leads to more substantial decline in summarization performance compared to shallower layers, highlighting the crucial role of deeper layers in integrating essential information. Notably, the study confirms the hypothesis that the hidden layers exhibiting higher similarity to human brain activity play a more critical role in model performance, with their correlations reaching statistical significance even after controlling for perplexity. These findings deepen our understanding of the cognitive mechanisms underlying language representations in DLMs and their neural correlates, potentially providing insights for optimizing and improving language models by aligning them with the human brain's language-processing mechanisms.