Related Experiment Video
Updated: Jan 16, 2026
The Participant-Reported Implementation Update and Score PRIUS: A Novel Method for Capturing Implementation-Related Data Over Time
Published on: February 19, 2021
The Unreliability of Two Publicly Reported Outcome Quality Measures for Characterizing Health Care Quality Within the
Kenneth J Nieser1,2, Daniel J Tancredi3, Alex H S Harris1,2
1VA Palo Alto Health Care System, Center for Innovation to Implementation, Menlo Park, California, USA.
Objective:
To estimate the reliability of two outcome quality measures in Veterans Health Administration (VHA) data using three different methods.
Study Setting And Design:
We created two cohorts of VHA patients meeting criteria for two measures: (1) risk-standardized complication rates following elective primary total hip arthroplasty and/or total knee arthroplasty (THA/TKA), and (2) risk-standardized mortality rates following acute myocardial infarction hospitalization (AMI). We fit hierarchical logistic regression models and calculated facility-level risk-standardized rates. We estimated entity-level reliability using three commonly applied methods: (1) delta method approximation; (2) latent scale model; (3) split-sample method.
Data Sources And Analytic Sample:
For each measure, we extracted risk adjustment and outcome data from the VHA Corporate Data Warehouse for patients meeting eligibility criteria in fiscal years 2021 and 2022.
Principal Findings:
Most facilities had complication rates following total hip and/or knee arthroplasty and mortality rates following hospitalization for acute myocardial infarction that, statistically, were no different from the national average. Reliability estimates based on delta method approximation (0.14 for THA/TKA; 0.12 for AMI) and the split-sample method (0.12 for THA/TKA; 0.19 for AMI) were very low for both measures. As we varied the sample sizes, we found that much higher sample sizes would be needed to reliably differentiate quality of care across facilities. On the other hand, reliability estimates based on the latent scale model were substantially higher than the other two methods (0.64 for THA/TKA; 0.41 for AMI), suggesting that there is substantially more between-facility variation in latent quality than manifests in observed outcomes.
Conclusions:
Reliability estimates based on the latent scale approach are not numerically or conceptually interchangeable with estimates based on the other two approaches. Given that health outcomes are generally reported using observed outcomes, reliability estimation based on the latent scale approach should not be used without a strong rationale.
More Related Videos
06:55Inverse Probability of Treatment Weighting Propensity Score using the Military Health System Data Repository and National Death Index
Published on: January 8, 2020
02:28Author Spotlight: Self-Assessment Protocol for Predicting Psoriatic Arthritis in Psoriasis Patients
Published on: March 1, 2024
Related Concept Videos
01:11Purpose of Health Records I
Here's a breakdown of how health records serve these purposes:
01:30Health Information Technology and Healthcare Information System
Health Information Technology, commonly called HIT, integrates advanced information systems and technology in healthcare settings. Its primary functions include:
01:25Errors occurring during blood pressure monitoring
Several factors...
01:11Guidelines for Writing Outcome
Patient outcomes reflect the patient's response to the goal rather than what the nurse aims to achieve. Terminology should be observable and measurable to avoid the reader's interpretation. The desired outcome should be realistic and achievable in the designated care timeframe. Expected outcomes should align with adjunctive therapies. The outcome should enhance care...
01:15Nursing Evaluation
01:29Documentation in Long-Term and Home Healthcare Setting
Long-Term Care Facilities