Inter-rater reliability of nursing home surveys: a Bayesian latent class approach

In the U.S., federal and state governments perform routine inspections of nursing homes. Results of the inspections allow government to generate fines for findings of non-compliance as well as allow consumers to rank facilities. The purpose of this study is to investigate the inter-rater reliability of the nursing home survey process. In general, the survey data involves 191 binary deficiency variables interpreted as 'deficient' or 'non-deficient'. To reduce the dimensionality of the problem, our proposed method involves two steps. First, we reduce the deficiency categories to sub-categories using previous nursing home studies. Second, looking at the State of Kansas specifically, we take the deficiency data from 1 year, and use Bayesian latent class analysis (LCA) to collapse the sub-categories to a binary variable. We evaluate inter-rater agreement using deficiency data from two separate survey teams on one facility, a matched-pair design. We evaluate the agreement of the two raters on binary data using the weights from the LCA. This allows a two-by-two contingency analysis using a Bayesian beta-binomial model. We elicit informative prior distributions from the nursing home providers. Together, with the experimental data, this provides a posterior distribution of the kappa agreement of the raters for nursing home deficiency citation data.