Expanding Integrity Measurement in Mental Health Care: A Coding System of Targets, Practices, and Steps

Service integrity, or the comparison of what has occurred with what was expected to occur, is essential to managing service quality. This paper tested the reliability of an expanded set of integrity metrics that included targets (i.e. problems targeted in intervention), practices (i.e. clinical procedures), and steps (i.e. sub-components of practices) measured at different levels of dialogue and across different types of service events.

This study was conducted in two publicly funded youth mental health systems located in geographically distinct, under-resourced communities where service inequities are common. Targets, practices, and steps were coded in 78 treatment events and 84 clinical supervision events that were digitally recorded and transcribed. Fleiss' kappa (K) and intraclass correlation coefficients (ICCs) yielded estimates for the interrater reliability of codes in excerpts (i.e. small contiguous units of dialogue) as well as complete events (i.e. entire supervision or treatment sessions).

Coders generally detected targets, practices, and steps in supervision and treatment with acceptable reliability. On average, coders detected steps more reliably than practices, targets more reliably in excerpts than in events, and supervision content more reliably than treatment content.

These findings demonstrate the potential to reliably code a uniform set of metrics for expanding the analysis of service integrity in dynamic care settings. Assessing the extent to which delivered practices match targets, delivered treatment matches supervision discussions, and delivered steps reflect the practices intended to be delivered can yield crucial insights for improving service coordination, continuity, and overall quality of mental health care.