在项目响应理论下建模评级订单效应 评级中介评估的评级模型
1Department of Psychology and Counseling, University of Taipei, Taipei, Taiwan.
Applied psychological measurement
|June 7, 2023
概括
新的项目响应理论 (IRT) 模型解决了评估中的动态评分器效应. 忽视评级顺序偏差可能会扭曲结果,影响评级者介导评估的准确性.
科学领域:
- 心理测量 心理测量 心理测量
- 教育测量教育的测量
- 统计建模 统计建模
背景情况:
- 在涉及人类判断的评估中,Rater效应很普遍.
- 现有的项目响应理论 (IRT) 模型主要解决静态评分器效应.
- 动态评级者效应,例如判断疲劳,可以随着时间的推移影响评级质量.
研究的目的:
- 开发新的多面 (MF) -IRT模型,考虑动态评级者效应,特别是评级顺序效应.
- 调查评级顺序对评级得分和评级者严重性的影响.
- 为分析评级者介导的评估提供一个框架,其中评级者表现可能发生变化.
主要方法:
- 开发了两个MF-IRT模型,其中包含系统或随机评级者严重性偏移.
- 在拟议的模型中使用贝叶斯估计来估计参数.
- 进行两项模拟研究以评估模型性能和参数恢复.
主要成果:
- 新开发的MF-IRT模型通过贝叶斯方法证明了通过贝叶斯方法令人满意的参数估计.
- 忽视评级顺序效应导致对模型结构和评级能力的偏见估计.
- 模拟证实了在IRT中考虑动态评分器效应的必要性.
结论:
- 拟议的MF-IRT模型有效地捕捉了像评级顺序效应这样的动态评级者效应.
- 如果不考虑评级顺序偏差,可能会严重损害评估结果的有效性.
- 这些模型提供了一个强大的方法来分析评级者介导的评估,提高准确性和公平性.
更多相关视频
07:43Author Spotlight: A Novel Setup to Conduct Naturalistic Laboratory Experiments with Real Human Actors in Scenarios
Published on: August 4, 2023
2.1K
09:00Author Spotlight: Validation of SICOLE-R for Assessing Cognitive and Reading Skills in Spanish-Speaking Children and Its Role in Personalized Education
Published on: August 16, 2024
829
相关概念视频
01:16Response Surface Methodology
191
Response Surface Methodology (RSM) is a collection of statistical and mathematical techniques used to develop, improve, and optimize processes. It is particularly valuable when many input variables or factors potentially influence a response variable.
The process of RSM involves several key steps:
The process of RSM involves several key steps:
191
02:16Stereotype Content Model
14.8K
The Stereotype Content Model (SCM) was first proposed by Susan Fiske and her colleagues (Fiske, Cuddy, Glick & Xu, 2002; see also Fiske, 2012 and Fiske, 2017). The SCM specifies that when someone encounters a new group, they will stereotype them based on two metrics: warmth—or that group’s perceived intent, and how likely they are to provide help or inflict harm—and competence—or their ability to carry out that objective. Depending on the warmth-competence...
14.8K
01:02Ranks
270
Unlike parametric methods, nonparametric statistics are ideal for nominal and ordinal data, requiring fewer assumptions about the population's nature or distribution. This makes nonparametric methods easier to apply and interpret, as they do not depend on parameters like mean or standard deviation. One common approach in nonparametric analysis is to sort data according to a specific criterion. For instance, we might arrange weather data from hottest to coldest days in a month or rank cities...
270
01:26Modeling in Therapy
128
Modeling, a key technique in therapy, uses observational learning to help clients acquire and practice new skills by watching therapists demonstrate desired behaviors. This approach, rooted in Albert Bandura's concept of vicarious learning, plays a significant role in therapeutic interventions for various psychological conditions, including social anxiety, ADHD, and depression.
Participant Modeling
Participant modeling involves therapists demonstrating calm and effective behaviors in...
Participant Modeling
Participant modeling involves therapists demonstrating calm and effective behaviors in...
128
01:21Friedman Two-way Analysis of Variance by Ranks
256
Friedman's Two-Way Analysis of Variance by Ranks is a nonparametric test designed to identify differences across multiple test attempts when traditional assumptions of normality and equal variances do not apply. Unlike conventional ANOVA, which requires normally distributed data with equal variances, Friedman's test is ideal for ordinal or non-normally distributed data, making it particularly useful for analyzing dependent samples, such as matched subjects over time or repeated measures...
256
00:54Ratio Level of Measurement
18.5K
The way a set of data is measured is called its level of measurement. Correct statistical procedures depend on a researcher being familiar with levels of measurement. For analysis, data are classified into four levels of measurement—nominal, ordinal, interval, and ratio.
A set of data measured using the ratio scale takes care of the ratio problem and provides complete information. Ratio scale data are like interval scale data, except they have a zero point and ratios can be calculated....
A set of data measured using the ratio scale takes care of the ratio problem and provides complete information. Ratio scale data are like interval scale data, except they have a zero point and ratios can be calculated....
18.5K