Using relative improvement over chance (RIOC) to examine agreement between tests: three case examples using studies...

Although statistics such as kappa and phi are commonly used to assess agreement between tests, in situations where the base rate of a disorder in a population is low or high, these statistics tend to underestimate actual agreement. This can occur even if the tests are good and the classification of subjects is adequate. Relative improvement over chance or RIOC is a statistic that can correct for this bias; however, it is not yet commonly used in the health sciences or disability fields. In this paper, we introduce RIOC and demonstrate its application using the results from 3 previously published studies, all of which assessed the agreement between tests commonly used to identify children with a neurodevelopmental disorder known as developmental coordination disorder (or DCD). The results illustrate the differences between kappa and RIOC under conditions where the distribution of scores in a 2 × 2 table is unbalanced. The introduction of this statistic in the area of developmental disabilities research is encouraged.