对于描述性相似性响应任务的自动评分的性能评估
Ryunosuke Oka1, Takashi Kusumi2, Akira Utsumi3
1Mitsubishi Electric Corporation, Kamakura, Kanagawa, 247-8501, Japan. Qualia1006@gmail.com.
Scientific reports
|March 15, 2024
概括
一个机器学习模型在日本语义相似性测试 (SST) 中获得了人类水平的性能. 自动化系统展示了与人类得分器相比较的高准确性和评分器间可靠性.
科学领域:
- 自然语言处理自然语言处理.
- 教育中的人工智能
- 心理测量 心理测量 心理测量
背景情况:
- 语义相似性测试 (SST) 评估对概念关系的理解.
- 自动评分系统越来越多地用于教育和心理评估.
- 评估机器学习模型与人类判断的性能对于它们的采用至关重要.
研究的目的:
- 调查机器学习模型在复制日本SST上人类得分的有效性.
- 将自动评分的准确性和可靠性与人类的表现进行比较.
主要方法:
- 来自变压器 (BERT) 模型的双向编码器表示得到了训练,使用SST的参与者响应和分类标准.
- 五倍交叉验证用于模型评估.
- 用准确性和评审者间可靠性指标评估模型性能.
主要成果:
- 该模型在对响应和分类标准进行训练时,达到83%的准确性 (实验1).
- 在一个单独的测试组中,该模型获得了80%的准确性 (实验2).
- 人类模型间评价者可靠性 (0.63) 与人与人之间的可靠性 (0.67-0.72) 非常接近.
结论:
- 机器学习模型可以在日本SST得分方面达到人类水平的性能.
- 自动评分系统为相似性任务提供了可靠的替代人类评分.
- 这项研究支持在语言和认知研究中使用人工智能进行客观和高效的评估.
相关概念视频
Response Surface Methodology
129
Response Surface Methodology (RSM) is a collection of statistical and mathematical techniques used to develop, improve, and optimize processes. It is particularly valuable when many input variables or factors potentially influence a response variable.
The process of RSM involves several key steps:
The process of RSM involves several key steps:
129
Wilcoxon Signed-Ranks Test for Matched Pairs
123
The Wilcoxon signed-rank test for matched pairs evaluates the null hypothesis by combining the ranks of differences with their signs. It essentially tests whether the median of the differences in a population of matched pairs is zero. Since the test incorporates more information than the sign test, it generally yields more trustable conclusions. This test also does not require the data to follow a normal distribution, but two conditions must be met for it to be applicable: (1) the data must...
123
Spearman's Rank Correlation Test
783
Spearman's rank correlation test, also known as Spearman's rho, is a nonparametric method for assessing the strength and direction of association between two variables. This test is particularly valuable when the data distribution is unknown or when the assumption of normality does not hold. Named after the English psychologist and statistician Dr. Charles Edward Spearman, it serves as the nonparametric counterpart to Pearson's correlation coefficient.
Spearman's test calculates...
Spearman's test calculates...
783
Review and Preview
7.4K
In statistics, several tools are used to interpret the data. Measures of central tendency represent the characteristics of the data, such as mean, median, and mode. Additionally, measures of variance like standard deviation and range are used to find the spread of data from the mean. Relative standing measures the distance between data locations. Commonly used measures of relative standings are percentile, z score, and quartiles.
Percentiles are a type of fractile that partition data into...
Percentiles are a type of fractile that partition data into...
7.4K


