为库尔德新闻问题和答案建立一个基准数据集
1Computer Science Department, College of Science, University of Halabja, Kurdistan Region, Halabja, Iraq.
Data in brief
|September 24, 2024
概括
本研究介绍了库尔德语新闻问答数据集 (KNQAD),这是自然语言处理研究的宝贵资源. 该数据集包含15002个库尔德语新闻段落,手工创建的问答对,有助于开发库尔德语AI模型.
科学领域:
- 自然语言处理自然语言处理.
- 计算语言学 计算语言学
- 信息检索 信息检索
背景情况:
- 为库尔德语等低资源语言开发NLP资源对于推进AI包容性至关重要.
- 库尔德语语言现有的数据集有限,阻碍了在问答 (QA) 和相关领域的研究.
研究的目的:
- 介绍库尔德语新闻问答数据集 (KNQAD),这是库尔德语QA研究的新型数据集.
- 为培训和评估库尔德语质量保证系统提供全面的资源.
- 分析库尔德人质量保证任务的复杂性.
主要方法:
- 使用ParsHub软件从各种库尔德新闻网站提取数据.
- 手动创建15002个新闻段落的问题和答案对.
- 数据预处理,包括清理,规范化,停止词删除和干扰.
- 分析问题类型分布和问题和答案之间的词汇相似性.
主要成果:
- 创建了KNQAD,包括15002个库尔德语新闻段落与相关的问答对.
- 详细分析数据集中的问题类型和固有的复杂性.
- 展示库尔德语NLP任务的预处理技术.
结论:
- 库尔德语NLP的KNQAD是一个显著的贡献,使得问答系统的进步成为可能.
- 该数据集有助于在质量保证的背景下研究库尔德语语言的语言细微差别和复杂性.
- 这些发现支持开发更复杂,更准确的库尔德语AI应用程序.
相关概念视频
Surveys
14.7K
Often, psychologists develop surveys as a means of gathering data. Surveys are lists of questions to be answered by research participants, and can be delivered as paper-and-pencil questionnaires, administered electronically, or conducted verbally. Generally, the survey itself can be completed in a short time, and the ease of administering a survey makes it easy to collect data from a large number of people.
14.7K
Archival Research
16.0K
Some researchers gain access to large amounts of data without interacting with a single research participant. Instead, they use existing records to answer various research questions. This type of research approach is known as archival research. Archival research relies on looking at past records or data sets to look for interesting patterns or relationships. For example, a researcher might access the academic records of all individuals who enrolled in college within the past ten years and...
16.0K
Cochran's Q Test
242
Cochran's Q Test is a nonparametric statistical test used to determine if there are potential differences in the outcomes of three or more related groups on a binary (yes/no) or dichotomous outcome. It is essentially an extension of the McNemar Test, which is limited to two related samples - Cochran's Q test can handle three or more related samples, making it more versatile in scenarios where subjects are measured under multiple conditions. The test statistic follows a Chi-Square...
242


