Effective number of subjects and number of raters for inter-rater reliability studies

A reliability study in which multiple raters evaluate multiple subjects was assumed in order to confirm the inter-rater reliability of rating scales. The two-way ANOVA model defining subjects and raters as random effects was applied, and the combination of the number of subjects and the number of raters that minimizes variance of the intraclass correlation coefficient (ICC) with a fixed total number of ratings (number of subjects x number of raters) was studied. The results revealed that the optimal combination depends on the relative ratio (between-rater variance/error variance), and that large number of raters brings the high precision of estimation of the ICC when the relative ratio was greater. It was concluded that when information concerning the relative ratio is unreliable, the use of identical numbers of raters and subjects minimizes variance when the relative ratio is substantially large and variance is maximized, providing a good study design from the viewpoint of 'maximin'.