The Fill-Mask Association Test (FMAT): Measuring propositions in natural language

Area of Science:

• Computational social science
• Natural language processing
• Psychological measurement

Background:

• Large language models (LLMs) offer new avenues for computational analysis of psychology.
• Existing methods often rely on associative or direct self-report measures.
• A propositional approach to psychological measurement in natural language is needed.

Purpose of the Study:

• Introduce the Fill-Mask Association Test (FMAT) as a novel method for psychological analysis.
• Leverage Masked Language Models (MLMs) for a propositional perspective on societal psychology.
• Establish the reliability and validity of FMAT across various psychological constructs.

Main Methods:

• Utilized Bidirectional Encoder Representations from Transformers (BERT) models.
• Computed semantic probabilities for words filling masked cloze sentences.
• Conducted 15 studies to assess FMAT's performance in predicting associations, measuring attitudes, and capturing stereotypes.

Main Results:

• FMAT demonstrated reliability and validity in predicting factual associations, measuring attitudes/biases, and capturing social stereotypes.
• Replicated established psychological findings (e.g., Implicit Association Test) and big-data text analyses.
• Showcased robustness across 12 BERT model variants and diverse text corpora.

Conclusions:

• FMAT provides a fine-grained, language-based method for psychological measurement.
• Substantiates a propositional (vs. associative) view of semantic representation in language.
• An R package is available to facilitate broader application of FMAT in research.