Related Experiment Video
Updated: Jul 2, 2025
05:37
An R-Based Landscape Validation of a Competing Risk Model
Published on: September 16, 2022
2.1K
A novel agreement statistic using data on uncertainty in ratings
Jarcy Zee1, Laura Mariani2, Laura Barisoni3
1University of Pennsylvania Perelman School of Medicine, Department of Biostatistics, Epidemiology, and Informatics, and Children's Hospital of Philadelphia, Philadelphia, PA, USA.
Journal of the Royal Statistical Society. Series C, Applied Statistics
|February 23, 2024
Summary
This study introduces a new agreement statistic to accurately measure inter-rater reliability by accounting for chance agreement. It uses rater uncertainty data, improving upon existing methods for fields like pathology.
Area of Science:
- Biostatistics
- Pathology
- Medical Informatics
Background:
- Existing agreement metrics often rely on assumptions about chance agreement that may not hold true.
- Pathologists' ratings of kidney biopsy descriptors illustrate limitations of current agreement estimation methods.
Purpose of the Study:
- To develop a novel agreement statistic that empirically accounts for chance agreement.
- To provide a more robust measure of reliability in subjective assessments.
Main Methods:
- Proposed a new agreement statistic incorporating empirical probability of chance agreement.
- Estimated chance agreement using additional data on rater uncertainty for each rating.
- Derived a standard error estimator for the novel statistic.
Main Results:
- Simulation studies demonstrated the proposed statistic's unbiasedness.
- The statistic accurately estimates the probability of agreement after chance correction.
- The method shows improved reliability in scenarios with violated assumptions.
Conclusions:
- The novel agreement statistic offers a more accurate approach to inter-rater reliability.
- Accounting for empirical chance agreement enhances the validity of agreement measures.
- This method is particularly valuable in fields with complex subjective ratings, such as diagnostic pathology.
Related Concept Videos
00:54Uncertainty: Confidence Intervals
4.1K
The confidence interval is the range of values around the mean that contains the true mean. It is expressed as a probability percentage. The interpretation of a 95% confidence interval, for instance, is that the statistician is 95% confident that the true mean falls within the interval. The upper and lower limits of this range are known as confidence limits. The confidence limits for the true mean are estimated from the sample's mean, the standard deviation, and the statistical factor...
4.1K
01:19Testing a Claim about Standard Deviation
2.5K
A complete procedure to test a claim about population standard deviation or population variance is explained here.
The hypothesis testing for the claim of population standard deviation (or variance) requires the data and samples to be random and unbiased. The population distribution also must be normal. There is no specific requirement on the sample size as the estimation is based on the chi-square distribution.
As a first step, the hypothesis (null and alternative) concerning the claim about...
The hypothesis testing for the claim of population standard deviation (or variance) requires the data and samples to be random and unbiased. The population distribution also must be normal. There is no specific requirement on the sample size as the estimation is based on the chi-square distribution.
As a first step, the hypothesis (null and alternative) concerning the claim about...
2.5K
01:11Statistical Analysis: Overview
6.6K
When we take repeated measurements on the same or replicated samples, we will observe inconsistencies in the magnitude. These inconsistencies are called errors. To categorize and characterize these results and their errors, the researcher can use statistical analysis to determine the quality of the measurements and/or suitability of the methods.
One of the most commonly used statistical quantifiers is the mean, which is the ratio between the sum of the numerical values of all results and the...
One of the most commonly used statistical quantifiers is the mean, which is the ratio between the sum of the numerical values of all results and the...
6.6K
01:09Empirical Method to Interpret Standard Deviation
5.2K
The empirical rule, also known as the three-sigma rule, allows a statistician to interpret the standard deviation in a normally distributed dataset. The rule states that 68% of the data lies within one standard deviation from the mean, 95% lies within two standard deviations from the mean, and 99.7% lies within three standard deviations from the mean. Additionally, this rule is also called the 68-95-99.7 rule.
This rule is used widely in statistics to calculate the proportion of data values...
This rule is used widely in statistics to calculate the proportion of data values...
5.2K
00:59Uncertainty: Overview
555
In analytical chemistry, we often perform repetitive measurements to detect and minimize inaccuracies caused by both determinate and indeterminate errors. Despite the cares we take, the presence of random errors means that repeated measurements almost never have exactly the same magnitude. The collective difference between these measurements - observed values - and the estimated or expected value is called uncertainty. Uncertainty is conventionally written after the estimated or expected value.
555
01:02Ranks
236
Unlike parametric methods, nonparametric statistics are ideal for nominal and ordinal data, requiring fewer assumptions about the population's nature or distribution. This makes nonparametric methods easier to apply and interpret, as they do not depend on parameters like mean or standard deviation. One common approach in nonparametric analysis is to sort data according to a specific criterion. For instance, we might arrange weather data from hottest to coldest days in a month or rank cities...
236