Related Experiment Video
Updated: Jun 10, 2026

The Participant-Reported Implementation Update and Score (PRIUS): A Novel Method for Capturing Implementation-Related Data Over Time
Published on: February 19, 2021
Identifying quality improvement intervention evaluations: is consensus achievable?
M S Danz1, L V Rubenstein, S Hempel
1RAND Corporation, Santa Monica, CA 90407, USA. mjsdanz@gmail.com
Background:
The diversity of quality improvement interventions (QIIs) has impeded the use of evidence review to advance quality improvement activities. An agreed-upon framework for identifying QII articles would facilitate evidence review and consensus around best practices.
Aim:
To adapt and test evidence review methods for identifying empirical QII evaluations that would be suitable for assessing QII effectiveness, impact or success.
Design:
Literature search with measurement of multilevel inter-rater agreement and review of disagreement.
Methods:
Ten journals (2005-2007) were searched electronically and the output was screened based on title and abstract. Three pairs of reviewers then independently rated 22 articles, randomly selected from the screened list. Kappa statistics and percentage agreement were assessed. 12 stakeholders in quality improvement, including QII experts and journal editors, rated and discussed publications about which reviewers disagreed.
Results:
The level of agreement among reviewers for identifying empirical evaluations of QII development, implementation or results was 73% (with a paradoxically low kappa of 0.041). Discussion by raters and stakeholders regarding how to improve agreement focused on three controversial article selection issues: no data on patient health, provider behaviour or process of care outcomes; no evidence for adaptation of an intervention to a local context; and a design using only observational methods, as correlational analyses, with no comparison group.
Conclusion:
The level of reviewer agreement was only moderate. Reliable identification of relevant articles is an initial step in assessing published evidence. Advancement in quality improvement will depend on the theory- and consensus-based development and testing of a generalizable framework for identifying QII evaluations.
Related Concept Videos
Quality Assurance
Nursing Evaluation
Section...
Nursing Interventions II: Selecting and Classifying the Nursing Interventions
Guidelines for Writing Outcome
Patient outcomes reflect the patient's response to the goal rather than what the nurse aims to achieve. Terminology should be observable and measurable to avoid the reader's interpretation. The desired outcome should be realistic and achievable in the designated care timeframe. Expected outcomes should align with adjunctive therapies. The outcome should enhance care evaluation by...
Nursing Process for Patient and Caregiver Teaching III: Evaluation and Documentation
Nurses can use several methods to evaluate patient outcomes. For example, oral questions can assess cognitive learning, patient...
Methods of Documentation III: PIE