在多面的拉什模型中,在评分设计和评分器配置下,重新审视人类和机器学习评分器的可靠性
Xingyao Xiao1, Richard J Patz2, Mark R Wilson3
1Graduate School of Education, Stanford University, Stanford, California, USA.
The British journal of mathematical and statistical psychology
|January 31, 2026
概括
机器学习 (ML) 评分为构建响应 (CR) 项目提供了一个可扩展的替代方案,但人类和ML评价者偏差显著影响能力估计可靠性. 谨慎的评分设计和偏差管理对于准确的结果至关重要.
科学领域:
- 教育测量的教育测量.
- 心理测量 心理测量 心理测量
- 教育中的人工智能
背景情况:
- 构建响应 (CR) 项目评估更高层次的技能,但面临着人类评分变化和成本的挑战.
- 机器学习 (ML) 提供了一个可扩展的解决方案,但其在评级者介导模型中的心理测量影响需要进一步调查.
研究的目的:
- 检查评分设计,评分者偏差,ML不一致性和模型规范对多种CR评估中的能力估计可靠性的影响.
- 为了比较不同估计模型在不同评分条件下的性能.
主要方法:
- 蒙特卡洛模拟用于操纵人类和ML评分器偏差,ML不一致性和得分密度 (完整,重叠,孤立).
- 对比了五种估计模型,包括部分信贷模型 (PCM) 和多面部分信贷模型 (MFPCM).
主要成果:
- 系统偏见,而不是随机不一致,被确定为能力估计错误的主要来源.
- 混合人类-ML评分提高了与无偏见或对立偏见的估计,但当偏见一致时,错误会加剧.
- 固定门的部分信贷模型 (PCM) 始终优于更复杂的模型,并且定CR项目稳定了MFPCM估计.
结论:
- 评分设计和偏差结构,而不是模型复杂性,决定了混合评分的有效性.
- 将构建响应项与选择响应指标挂是提高ML辅助评分估计稳定性的实用策略.
相关概念视频
Reliability and Validity
13.9K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
13.9K
Controller Configurations
378
Controller configurations are crucial in a car's cruise control system because they manage speed over time to maintain a consistent pace regardless of road conditions, thereby meeting design goals. In traditional control systems, fixed-configuration design involves predetermined controller placement. System performance modifications are known as compensation.
Control-system compensation involves various configurations, most commonly series or cascade compensation, in which the controller...
Control-system compensation involves various configurations, most commonly series or cascade compensation, in which the controller...
378
Electron Configurations
26.2K
Electron configurations and orbital diagrams can be determined by applying the Aufbau principle (each added electron occupies the subshell of lowest energy available), Pauli exclusion principle (no two electrons can have the same set of four quantum numbers), and Hund’s rule of maximum multiplicity (whenever possible, electrons retain unpaired spins in degenerate orbitals).
The relative energies of the subshells determine the order in which atomic orbitals are filled (1s, 2s, 2p, 3s, 3p,...
The relative energies of the subshells determine the order in which atomic orbitals are filled (1s, 2s, 2p, 3s, 3p,...
26.2K
Simplified Synchronous Machine Model
778
The Synchronous Machine Model is a fundamental tool in analyzing and ensuring the transient stability of power systems. This model simplifies the representation of a synchronous machine under balanced three-phase positive-sequence conditions, assuming constant excitation and ignoring losses and saturation. The model is pivotal for understanding the behavior of synchronous generators connected to a power grid, particularly during transient events.
In this model, each generator is connected to a...
In this model, each generator is connected to a...
778
Wind Turbine Machine Models
603
In the growing field of wind energy, incorporating wind turbine models into transient stability analysis is essential. Induction and synchronous machines are the primary models used, with induction machines being prevalent due to their simplicity and reliability.
Induction machines interact through the rotating magnetic field generated by the stator and the rotor. The key parameter is slip, which is the difference between synchronous speed and rotor speed relative to synchronous speed. Slip is...
Induction machines interact through the rotating magnetic field generated by the stator and the rotor. The key parameter is slip, which is the difference between synchronous speed and rotor speed relative to synchronous speed. Slip is...
603
Electron Configuration of Multielectron Atoms
64.9K
The alkali metal sodium (atomic number 11) has one more electron than the neon atom. This electron must go into the lowest-energy subshell available, the 3s orbital, giving a 1s22s22p63s1 configuration. The electrons occupying the outermost shell orbital(s) (highest value of n) are called valence electrons, and those occupying the inner shell orbitals are called core electrons. Since the core electron shells correspond to noble gas electron configurations, we can abbreviate electron...
64.9K


