Language-based affect assessments capture experiment-induced changes beyond rating scales

Language-based assessments (LBAs) have shown strong convergent and criterion validity cross-sectionally and longitudinally. However, no study has tested whether LBAs of affect can capture experiment-induced changes (i.e., causal validity). In this randomized mixed-design experiment, participants (N = 892) completed a mood induction procedure comprising different conditions (church, shopping mall, and park) using physical locations (N = 153) and online video clips (N = 739). Participants reported their pre- and post-induction affect through open-ended responses and ratings (i.e., the Positive and Negative Affect Schedule; PANAS). Using large language models and machine learning, we compared how accurately PANAS and language responses classified the experimental conditions. The LBA derived from the language responses exhibited significantly higher accuracy compared to the PANAS scores in classifying the conditions in both the training set (cross-validated AUC = .74 vs. .63), the online holdout set (AUC = .76 vs. .70), and the offline holdout set (AUC = .67 vs. .53). Apart from showing higher predictive accuracy, we assessed LBAs of specific affective dimensions (valence, arousal, and basic emotions). These analyses revealed that positive valence was most strongly expressed in the park condition, while anger was particularly elevated in the mall condition. We also visualized language differences across conditions, showing more calm and peaceful expressions in the park condition and more tired and bored expressions in the mall condition. These findings support the causal validity of language-based affect assessments and their ability to provide insights into affective states. We discuss the potential of using LBA in experimental psychological research.