Related Experiment Video
Updated: Jun 8, 2026
08:40
Isokinetic Robotic Device to Improve Test-Retest and Inter-Rater Reliability for Stretch Reflex Measurements in Stroke Patients with Spasticity
Published on: June 12, 2019
Comparison between inter-rater reliability and inter-rater agreement in performance assessment
Shih Chieh Liao1, Elizabeth A Hunt, Walter Chen
1School of Medicine, College of Medicine, China Medical University, Taichung, Taiwan.
Annals of the Academy of Medicine, Singapore
|September 15, 2010
Summary
Inter-rater agreement and inter-rater reliability in performance assessment (PA) are not guaranteed to coexist. This study demonstrates that one can be high while the other is low, highlighting their distinct roles in ensuring accurate scoring consistency.
Area of Science:
- Medical Education
- Assessment and Evaluation
Background:
- Performance assessment (PA) is crucial in medical education, with Objective Structured Clinical Examination (OSCE) as a prime example.
- Accurate PA relies on scoring consistency among multiple raters.
- Inter-rater agreement and inter-rater reliability are key indices for ensuring scoring consistency.
Purpose of the Study:
- To investigate the relationship between inter-rater agreement and inter-rater reliability in performance assessment.
- To clarify the distinct contributions of agreement and reliability to scoring consistency.
Main Methods:
- Utilized three distinct sets of simulated data.
- Data simulated raters' evaluations of student performance.
- Analyzed the interplay between inter-rater agreement and inter-rater reliability.
Main Results:
- Dataset 1: High inter-rater agreement, low inter-rater reliability.
- Dataset 2: High inter-rater reliability, low inter-rater agreement.
- Dataset 3: High inter-rater agreement and high inter-rater reliability.
Conclusions:
- Inter-rater agreement and inter-rater reliability do not necessarily coexist; one does not guarantee the other.
- Both measures are vital for robust PA.
- Inter-rater agreement reflects score stability across raters for a single student, while inter-rater reliability indicates score consistency across students evaluated by different raters.
Related Concept Videos
01:20Kendall's Coefficient of Concordance
Kendall's Coefficient of Concordance (W), also known as Kendall's W, is a non-parametric statistical measure used to assess the agreement or concordance between multiple raters or judges when they rank a set of items. It is often used when you have ordinal data (ranks) and you want to see if there is consistency or consensus among the raters. It is widely applied in research areas such as psychology, medicine, and social sciences, where multiple judges are asked to rank or rate subjects or...
01:29Reliability and Validity
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
01:20Spearman's Rank Correlation Test
Spearman's rank correlation test, also known as Spearman's rho, is a nonparametric method for assessing the strength and direction of association between two variables. This test is particularly valuable when the data distribution is unknown or when the assumption of normality does not hold. Named after the English psychologist and statistician Dr. Charles Edward Spearman, it serves as the nonparametric counterpart to Pearson's correlation coefficient.
Spearman's test calculates correlation by...
Spearman's test calculates correlation by...
01:10Calibration Curves: Correlation Coefficient
In a linear calibration curve, there is a value called the calibration coefficient, denoted by 'r,' which measures the strength and the direction of association between two variables. The correlation coefficient value ranges from −1 to +1. A value of +1 indicates a perfect positive linear correlation, −1 denotes a perfect negative correlation, and 0 implies no correlation between the two variables. A positive correlation value establishes that as one variable increases, the other increases, and...
01:09Comparing Experimental Results: Student's t-Test
The t-test is a statistical method used to compare the sample mean with a population mean or compare two means from two data sets. The test statistic is calculated from the standard deviation, mean, and number of measurements in the data set at a selected confidence interval and then compared to a table of critical values at this confidence level. If the test statistic is smaller than the critical value, the null hypothesis is accepted. In this case, we state that the difference between the...
00:54Ratio Level of Measurement
The way a set of data is measured is called its level of measurement. Correct statistical procedures depend on a researcher being familiar with levels of measurement. For analysis, data are classified into four levels of measurement—nominal, ordinal, interval, and ratio.
A set of data measured using the ratio scale takes care of the ratio problem and provides complete information. Ratio scale data are like interval scale data, except they have a zero point and ratios can be calculated. For...
A set of data measured using the ratio scale takes care of the ratio problem and provides complete information. Ratio scale data are like interval scale data, except they have a zero point and ratios can be calculated. For...