Evaluating reliability. A cost-effectiveness approach

This study derives the economic costs of misclassification in nursing home patient classification systems. These costs are then used as weights to estimate the reliability of a functional assessment instrument applied to 290 Veterans Administration nursing home patients. The weighted kappa is used to measure reliability and the results were generally favorable. Use of the unweighted kappa indicated significant reliability problems. It is suggested that reliability must be redefined and remeasured with each substantively new application of an assessment instrument.