从协议到认识体系对齐:一个信号检测理论模型的Inter-Rater可靠性
1Free University of Bozen-Bolzano, Italy.
Educational and psychological measurement
|February 19, 2026
概括
科恩科恩科恩科恩科恩科恩
科学领域:
- 心理学 心理学 心理学
- 统计 统计 统计 统计
- 认知科学 认知科学
背景情况:
- 评价者之间的可靠性通常使用科恩的卡帕 (κ) 来衡量,它量化了协议,但没有建模判断过程.
- 科恩的卡帕对患病率,难度和不同的评分标准敏感,导致对诊断准确性的误解.
研究的目的:
- 使用信号检测理论 (SDT) 重构分级间的可靠性.
- 引入战略收指数 (SCI),以衡量评级者在SDT框架中的决策门对齐.
主要方法:
- 为分类判断开发了一个信号检测理论生成模型.
- 引入战略收指数 (SCI) 来量化评级者决策门的收.
- 在不同的条件下进行蒙特卡洛模拟,以比较科恩的卡帕和SCI.
主要成果:
- 科恩的卡帕因患病率和可歧视性而有所不同,即使在持续的决策政策下也是如此.
- 战略收指数 (SCI) 选择性地衡量认识体系对齐,并且对这些因素不变.
- 即使对其真实状态的不确定性,SCI也可以作为稳定的财产被收回.
结论:
- 科恩的卡帕是战略变异的有限转换,而不是认识体系对齐的直接衡量.
- 战略收框架将可靠性分解为结果协议和流程调整.
- 该框架通过将统计数据与生成模型联系起来,将可靠性评估从描述向解释推进.
相关概念视频
Understanding Deception
181
Deception is a pervasive aspect of human communication. Empirical studies have shown that most individuals engage in some form of deceit on a daily basis, with approximately 20% of social exchanges involving deceptive elements. Lying follows a developmental trajectory, peaking during adolescence and declining with age, possibly due to the maturation of cognitive control and social accountability.Cognitive and Social Factors in Deception DetectionDespite its prevalence, accurately detecting...
181
Calibration Curves: Correlation Coefficient
5.0K
In a linear calibration curve, there is a value called the calibration coefficient, denoted by 'r,' which measures the strength and the direction of association between two variables. The correlation coefficient value ranges from −1 to +1. A value of +1 indicates a perfect positive linear correlation, −1 denotes a perfect negative correlation, and 0 implies no correlation between the two variables. A positive correlation value establishes that as one variable increases, the...
5.0K
Kendall's Coefficient of Concordance
1.1K
Kendall's Coefficient of Concordance (W), also known as Kendall's W, is a non-parametric statistical measure used to assess the agreement or concordance between multiple raters or judges when they rank a set of items. It is often used when you have ordinal data (ranks) and you want to see if there is consistency or consensus among the raters. It is widely applied in research areas such as psychology, medicine, and social sciences, where multiple judges are asked to rank or rate subjects...
1.1K
Theory of Attribution I: Correspondent Inference Theory
627
Correspondent inference theory, proposed by Jones and Davis in 1965, seeks to explain how individuals infer stable personality traits from observed behaviors. It suggests that people attribute actions to underlying dispositions rather than external circumstances, particularly when the behavior appears intentional and socially significant.Voluntary Behavior and Dispositional AttributionAccording to this theory, individuals are more likely to attribute behavior to personal traits when it appears...
627
Reliability and Validity
14.2K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
14.2K
Statistical Analysis: Overview
16.7K
When we take repeated measurements on the same or replicated samples, we will observe inconsistencies in the magnitude. These inconsistencies are called errors. To categorize and characterize these results and their errors, the researcher can use statistical analysis to determine the quality of the measurements and/or suitability of the methods.
One of the most commonly used statistical quantifiers is the mean, which is the ratio between the sum of the numerical values of all results and the...
One of the most commonly used statistical quantifiers is the mean, which is the ratio between the sum of the numerical values of all results and the...
16.7K

