Rater reliability in reading PA films of hands for bone and cartilage changes in rheumatoid arthritis

Three experiments were conducted to evaluate the rater reliability for scoring osseous defects (articular erosions) and joint space narrowing (cartilage destruction) in patients with rheumatoid arthritis. The experiments show that two raters, using different techniques, obtain similar scores; the scores differing primarily by a multiplicative factor. A single rater achieves reproducible results under similar conditions. But with a change in reading technique, such as using a magnifying lens and intense light, the same rater obtains a change in scores. The problems of measurement must be carefully examined for any proposed measurement technique. The superficial objectivity and precision of a measurement scale are less important than its objectively and precision of a measurement scale are less important than its objectively evaluated reliability. Comparing alternative procedures is a statistical problem, and can be subjected to suitable statistical analysis to evaluate alternatives.