Optimal two-stage reliability studies

The intraclass correlation is often used to assess the reliability of a measurement system. There is a considerable literature devoted to optimizing the standard assessment plan in which a number of subjects are measured repeatedly. We propose a two-stage investigation, here called a leveraged plan (LP), where in Stage I, we measure a number of subjects once. Then in Stage II, we select a subset of subjects with extreme initial measurements for repeated measurement. For a fixed total number of measurements, we show that the optimal LP provides a more precise estimate of the intraclass correlation coefficient than does the optimal standard plan (SP). We provide a table for finding the optimal LP given the true intraclass correlation and a specified precision for the estimate. For a fixed total number of measurements N, a nearly optimal LP makes roughly N/2 measurements in Stage I and then selects roughly N/6 extreme subjects to re-measure thrice each in Stage II. We also compare optimal leveraged with optimal SPs when there is a limit on the number of times each subject can be re-measured.