创建和验证细粒度问题主观性数据集 (FQSD):增强自动主观问题答案系统的新基准
Marzieh Babaali1, Afsaneh Fatemi1, Mohammad Ali Nematbakhsh1
1Faculty of Computer Engineering, University of Isfahan, Isfahan, Iran.
PloS one
|May 23, 2024
概括
一个新的细粒度问题主观性数据集 (FQSD) 包含10,000个问题,可以推进自动主观问题答案系统. 这一数据集提供了详细的主观性和比较形式的分类,实现了高度的注释可靠性.
科学领域:
- 自然语言处理自然语言处理.
- 人工智能的人工智能
- 计算语言学 计算语言学
背景情况:
- 现有的数据集缺乏详细的问题主观性分类,阻碍了自动主观问题答案 (ASQA) 系统的进步.
- 需要全面的数据集来提高问题中主观性检测的准确性和可靠性.
研究的目的:
- 介绍细粒度问题主观性数据集 (FQSD),这是一个用于问题主观性分类的新资源.
- 提供1万个问题的详细数据集,按主观性 (客观/主观) 和细粒度类型 (目标,态度,理由,是/否,无) 和比较形式 (单个,比较) 分类.
主要方法:
- 开发并注释了一个由10,000个问题组成的数据集,使用多个注释器.
- 使用弗莱斯卡帕 (0.76) 和皮尔森相关性 (高达0.80) 验证了注释可靠性.
- 将FQSD与现有数据集进行比较,并使用变压器模型 (BERT,XLNET,RoBERTa) 进行验证,并使用LIME进行模型解释.
主要成果:
- 与现有的数据集相比,FQSD在规模,语言多样性和语法复杂性方面表现出色.
- 罗伯塔在使用FQSD的主观性分类上获得97%的F1分数,证明了其有效性.
- LIME分析证实了主观性分类的透明和可靠的模型预测.
结论:
- FQSD是推动ASQA系统和问题主观性分类研究的宝贵资源.
- 数据集的质量和变压器模型的高性能验证了它对于复杂的NLP任务的实用性.
- 视觉方法和可解释性技术提高了主观性分类模型的理解和可靠性.
相关概念视频
Surveys
14.7K
Often, psychologists develop surveys as a means of gathering data. Surveys are lists of questions to be answered by research participants, and can be delivered as paper-and-pencil questionnaires, administered electronically, or conducted verbally. Generally, the survey itself can be completed in a short time, and the ease of administering a survey makes it easy to collect data from a large number of people.
14.7K
Stereotype Content Model
14.7K
The Stereotype Content Model (SCM) was first proposed by Susan Fiske and her colleagues (Fiske, Cuddy, Glick & Xu, 2002; see also Fiske, 2012 and Fiske, 2017). The SCM specifies that when someone encounters a new group, they will stereotype them based on two metrics: warmth—or that group’s perceived intent, and how likely they are to provide help or inflict harm—and competence—or their ability to carry out that objective. Depending on the warmth-competence...
14.7K
Traits, Mood, and Subjective Wellbeing
69
Subjective well-being (SWB) refers to an individual's self-evaluation of their overall life satisfaction, happiness, and fulfillment. This multifaceted construct is typically assessed by analyzing the balance of positive and negative emotions alongside perceptions of life satisfaction. Personality traits such as neuroticism and extraversion are strongly associated with variations in SWB, offering critical insights into the underlying mechanisms of emotional well-being.
Neuroticism and...
Neuroticism and...
69
Systematic Error: Methodological and Sampling Errors
1.5K
In the case of systematic errors, the sources can be identified, and the errors can be subsequently minimized by addressing these sources. According to the source, systematic errors can be divided into sampling, instrumental, methodological, and personal errors.
Sampling errors originate from improper sampling methods or the wrong sample population. These errors can be minimized by refining the sampling strategy. Defective instruments or faulty calibrations are the sources of instrumental...
Sampling errors originate from improper sampling methods or the wrong sample population. These errors can be minimized by refining the sampling strategy. Defective instruments or faulty calibrations are the sources of instrumental...
1.5K
Data Collection by Survey
6.4K
The systematic method of obtaining and analyzing accurate information of a population is called data collection. A survey is a standard method of data collection that involves collecting information from a target human population about their experience, opinion, or knowledge of a product, service, or process. The responses are recorded and interpreted. The most common survey examples are written questionnaires, face-to-face or telephonic conversations, focus groups, and electronic (e-mail or...
6.4K
Naturalistic Observations
15.4K
If you want to understand how behavior occurs, one of the best ways to gain information is to simply observe the behavior in its natural context. However, people might change their behavior in unexpected ways if they know they are being observed. How do researchers obtain accurate information when people tend to hide their natural behavior? As an example, imagine that your professor asks everyone in your class to raise their hand if they always wash their hands after using the restroom. Chances...
15.4K


