对短文本的文本挖掘方法的系统评估:从在线帖子绘制个人内部状态的地图
Ana Macanovic1, Wojtek Przepiorka2
1Department of Sociology/ICS, Utrecht University, Utrecht, The Netherlands. a.macanovic@uu.nl.
Behavior research methods
|April 4, 2024
概括
在人类编码数据上训练的大型语言模型最好分析在线文本以获得社会科学见解. 更简单的方法和像GPT-4这样的先进模型显示出潜力,但并不总是与这种性能相匹配.
科学领域:
- 社会和行为科学 社会和行为科学
- 计算社会科学 计算社会科学
- 自然语言处理自然语言处理.
背景情况:
- 在线文本提供了有关个体内部状态的丰富数据.
- 人类专家的手动编码是可靠的,但限制了数据规模.
- 对于大规模的文本数据,需要自动化文本分析方法.
研究的目的:
- 评估社会科学的自动文本分析方法.
- 在四个编码任务中与训练有素的人类编码器进行性能比较.
- 确定分析大型文本数据集内部状态的最佳方法.
主要方法:
- 比较基于字典的方法,训练有素的大型语言模型 (LLM) 和使用GPT-4进行零射击分类.
- 在编码动机,规范,情绪和立场的表达方面进行评估的表现.
- 评估了不同文本分析方法的准确性和错误阳性率.
主要成果:
- 在手动编码数据上训练的LLM在所有任务中获得了最高的性能.
- 字典方法经常产生错误的阳性结果,尽管识别了不常见的类别.
- GPT-4的零射击分类显示出有希望的结果,但受过训练的LLMs表现不佳.
结论:
- 经过培训的LLM是在线文本中对内部状态进行大规模分析的最有效方法.
- 更简单的方法可以在特定情况下有效,在复杂性和性能之间提供权衡.
- 社会科学家在选择文本分析工具时应考虑模型性能和特定应用需求.
更多相关视频
08:53Integrating Computerized Linguistic and Social Network Analyses to Capture Addiction Recovery Capital in an Online Community
Published on: May 31, 2019
5.2K
09:20Cloud-Based Phrase Mining and Analysis of User-Defined Phrase-Category Association in Biomedical Publications
Published on: February 23, 2019
8.7K
相关概念视频
Empathy
Some researchers suggest that altruism operates on empathy. Empathy is the capacity to understand another person’s perspective, to feel what he or she feels. An empathetic person makes an emotional connection with others and feels compelled to help (Batson, 1991). Empathy can be expressed in several ways, including cognitive, affective, and motor.
Self-Evaluation Maintenance Model
The Self-Evaluation Maintenance (SEM) model offers a psychological framework to understand how individuals’ self-esteem is influenced by the achievements of others, particularly those with whom they share close personal bonds. The SEM model operates when personal rather than social identity guides individuals. Central to this model is the notion that individuals have an inherent desire to preserve a favorable self-image, which is continuously shaped by interpersonal comparisons and...
Introspection
Introspection, long upheld as a reliable route to self-knowledge, involves examining one's thoughts, emotions, and mental processes. It underpins many psychological practices, from mindfulness meditation to psychotherapy and self-help strategies. However, empirical evidence challenges the accuracy of introspection as a means of understanding oneself.Limitations of Introspective InsightSeminal work by Nisbett and Wilson demonstrated that individuals are frequently unaware of the true causes...
Self-Awareness and Its Effects
Self-awareness is a psychological state in which the individual becomes the focal point of their attention. This inward focus transforms the self into an object of contemplation and assessment, influencing how individuals perceive their actions and their alignment with personal and societal standards.Triggers and Contexts for Self-AwarenessSelf-awareness can be activated by external stimuli that make individuals visually or audibly aware of themselves, such as mirrors, cameras, or recordings.
