New knowledge source pipelines for sociocultural representations in a cognitive architecture

With the increasing use of large language models (LLMs) and related scaled knowledge systems, it is important to achieve a better picture of the downstream impacts of the ubiquity of these systems. LLMs (as they currently are instantiated) largely output text using the skewed training data and patterns recognized through pretraining. The datasets of these systems are collected primarily from WEIRD (Western, Educated, Industrialized, Rich, Democratic) societies, and those data are often drawn from sources that represent marginalized, exploited communities (in WEIRD societies) problematically. This data process renders uneven representation, skewing potential cultural insights, and the complicated structure of LLMs can cloud the task of elucidating the full extent of problematic sociocultural structure representation within those models.

Using LLMs with process-based computational cognitive models and architectures (e.g., using ACT-R) provides a potential avenue for understanding the implications of these systems on human cognitive processes. We present two systems that combine the ACT-R architecture with more scalable, holographic (vector-symbolic) memory representations - one using an LLM and another using the ConceptNet semantic network - and compare the two systems using a cognitive model of the implicit association task (IAT).

Though both systems are useful, we found that the ConceptNet based system more closely resembled human data from the IAT in its impact of sociocultural structure representation on behavior through memory. Nonetheless, both systems represent an opportunity to probe and evaluate those knowledge systems (i.e., ConceptNet and Llama3.2 in this case) using more cognitively relevant and plausible methods that are more generalizable to actual human behavior.

Though we present a model of a relatively simple task here in the IAT, the results point to a unique and powerful opportunity to develop a deeper understanding of what the ubiquity of these (and related) systems means for the propagation and bootstrapping of existing sociocultural knowledge and structures - especially those structures that uphold problematic, racialized representations of the human.