消除歧义的情感注释:混合方法调查注释者经验和指示对注释者协议的影响
Laura E M Äyräväinen1, Joanne Hinds1, Brittany I Davidson1
1School of Management, University of Bath, Bath, United Kingdom.
PloS one
|December 1, 2025
概括
详细的说明并不总是改善情绪分析数据集的人类注释一致性. 任务参与和直观策略显著影响注释器协议,影响自然语言处理模型的开发.
科学领域:
- 自然语言处理自然语言处理.
- 计算语言学 计算语言学
- 数据注释数据注释
背景情况:
- 人类注释的数据集对于开发和评估情绪分析和NLP系统至关重要.
- 现有的数据集往往具有较低的注释器协议和错误,损害了数据质量.
- 提高注释可靠性需要了解数据集是如何创建的,以及注释者的解释.
研究的目的:
- 介绍AmbiSent,这是一个新的情绪数据集,旨在解决解释复杂性的问题.
- 调查注释说明对注释者的经验和同意的影响.
- 分析影响人类注释数据集注释一致性的因素.
主要方法:
- 采用混合方法方法,将定量和定性分析结合起来.
- 两组群众工作者 (每组53人) 标注了252个句子,附有详细或最小的说明.
- 采用统计测试 (变换,独立样本t测试) 和反射主题分析来对调查答案进行分析.
主要成果:
- 仅仅是详细的说明并没有持续地改善标注者之间的或内部的协议.
- 标注的有效性取决于参与者的参与和与直观策略的结合.
- 特定的句子类型 (例如,视角依赖的情绪,修辞问题) 显示了与详细指令更好的协议的潜力.
结论:
- 标注指令的有效性受到任务参与和直观的标注方法的调节.
- 尽管有详细的指导,但仍观察到减少注释策略.
- 在未来的数据集开发中提供了建议,以加强任务参与和遵守指令.
- 环境数据集以三种版本发布,以支持各种NLP研究需求.
相关概念视频
02:01Group Design
10.1K
The most basic experimental design involves two groups: the experimental group and the control group. The two groups are designed to be the same except for one difference— experimental manipulation. The experimental group gets the experimental manipulation—that is, the treatment or variable being tested—and the control group does not. Since experimental manipulation is the only difference between the experimental and control groups, we can be sure that any differences between...
10.1K
01:29The Influence of Affect on Cognition
248
Positive affect significantly influences cognitive processes, including evaluation, memory, creativity, and social judgments. Compared to negative affect, positive emotional states promote more favorable interpretations of stimuli, cognitive flexibility, and heuristic processing. These effects highlight emotions' powerful role in shaping how individuals perceive, remember, and interact with the world.Influence on Evaluation and AttributionWhen individuals experience positive affect, they are...
248
01:50Statistical Significance
21.0K
Once data is collected from both the experimental and the control groups, a statistical analysis is conducted to find out if there are meaningful differences between the two groups. A statistical analysis determines how likely any difference found is due to chance (and thus not meaningful). In psychology, group differences are considered meaningful, or significant, if the odds that these differences occurred by chance alone are 5 percent or less. Stated another way, if we repeated this...
21.0K
01:20Cognitive Theories: Schachter-Singer Theory of Emotion
1.4K
Stanley Schachter and Jerome Singer proposed the two-factor theory of emotion, which emphasizes the interplay between physiological arousal and cognitive labeling in forming emotional experiences. This theory suggests that emotions are not simply a result of physiological responses but rather a combination of these responses and the individual's cognitive interpretation of them.
Physiological Arousal and Cognitive Labeling
According to this theory, when an individual experiences...
Physiological Arousal and Cognitive Labeling
According to this theory, when an individual experiences...
1.4K
02:16Surveys
16.6K
Often, psychologists develop surveys as a means of gathering data. Surveys are lists of questions to be answered by research participants, and can be delivered as paper-and-pencil questionnaires, administered electronically, or conducted verbally. Generally, the survey itself can be completed in a short time, and the ease of administering a survey makes it easy to collect data from a large number of people.
16.6K
02:30Naturalistic Observations
16.9K
If you want to understand how behavior occurs, one of the best ways to gain information is to simply observe the behavior in its natural context. However, people might change their behavior in unexpected ways if they know they are being observed. How do researchers obtain accurate information when people tend to hide their natural behavior? As an example, imagine that your professor asks everyone in your class to raise their hand if they always wash their hands after using the restroom. Chances...
16.9K