Sentiment Analysis of Naturalistic Speech Using Open-Weight Large Language Models

Area of Science:

• Computational Linguistics
• Psychological Science
• Artificial Intelligence

Background:

• Psychological research increasingly uses computational methods for emotion analysis in text.
• Standard lexicon tools lack semantic nuance, and commercial Large Language Models (LLMs) pose privacy risks for sensitive data.

Purpose of the Study:

• To evaluate the effectiveness of 24 open-weight LLMs for zero-shot sentiment analysis of spoken language on local hardware.
• To compare LLM performance against established baselines and assess privacy-preserving capabilities.

Main Methods:

• Tested 24 open-weight LLMs (1B-120B parameters) in a zero-shot setting for sentiment analysis.
• Compared model performance against naive, standard lexicon, and human baselines using two datasets of spoken narratives.
• Validated a privacy-preserving pipeline, assessing the impact of automatic speech recognition errors.

Main Results:

• Open-weight LLMs significantly outperformed lexicon-based tools and often surpassed human raters.
• Mid-sized models demonstrated performance comparable to larger systems, enhancing accessibility.
• Privacy-preserving pipeline showed minimal degradation in sentiment accuracy despite transcription errors.

Conclusions:

• Open-weight LLMs provide efficient, secure, and high-performance sentiment analysis for naturalistic speech on local hardware.
• These models offer promising avenues for studying emotional dynamics and developing privacy-preserving clinical tools.
• Fairness audits revealed demographic disparities, highlighting areas for future model development and refinement.