Related Experiment Video
Updated: Apr 30, 2026
Qualitative and Quantitative Validation of Tools with Rating Scales Aimed at Assessing the Quality of University Service-Learning
Published on: August 29, 2025
Examining the Validity of the Implementation Quality Rubric for Simulation (IQR-SIM) for Assessing Implementation
Leizl Joy Nayahangan1, Adam Dubrowski, Krystina Clarke
1Copenhagen Academy for Medical Education and Simulation (CAMES), The Capital Region of Denmark (L.J.N., L.K., A. G.); Department of Clinical Medicine, Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark (L.J.N., L.K., A.G.); maxSIMhealth Laboratory, Faculty of Health Sciences, Ontario Tech University, Oshawa (A.D., K.C.); Simulation Program, Unity Health Toronto (L.B.); Department of Physical Therapy, Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada (L.B.); NORDSIM, Aalborg University Hospital, Aalborg, Denmark (M.L.F.); Department of Medical Education, University of Illinois College of Medicine, IL (R.Y.).
Introduction:
The lack of reliable tools for measuring implementation processes and outcomes weakens confidence in study findings and strategy effectiveness. This study aimed to gather validity evidence for the previously published Implementation Quality Rubric for Simulation (IQR-SIM) from 2 sources: response process and internal structure.
Methods:
The IQR-SIM is structured into 3 phases of implementation, each divided into themes. Within these themes are items representing actionable steps for effective implementation. It employs a 3-point rating scale (1, 2, and 3), along with nonscored options "don't know/can't assess" and "not applicable." Semistructured interviews were conducted to collect detailed program descriptions from international simulation centers. Three trained raters evaluated these interview-derived implementation data using the IQR-SIM. The frequency of using the scores and the nonscored options was calculated to evaluate rater use of the scale. Inter-rater reliability was assessed to examine variability in ratings, while internal consistency reliability was analyzed to determine whether all items measured the same construct.
Results:
Twenty simulation-based programs from 7 international simulation centers participated. Phase 1 (stakeholder engagement and context exploration) had the highest proportion of fully accomplished items (54.6% received a rating of 3/3). Theme 1b (Obtaining buy-in from relevant stakeholders) was predominantly rated as fully accomplished (68.3%). Item 9 (planned for human resource needs) received the highest proportion of rating 3/3 (83.3%). Internal consistency reliability was good (Cronbach alpha=0.85). Inter-rater reliability analysis showed a moderate intraclass correlation coefficient (ICC=0.57). ICC values between raters ranged from 0.39 to 0.52.
Conclusions:
Multisite validity evidence regarding the response process and internal structure of the IQR-SIM was established, supporting its use in evaluating the implementation quality of simulation-based programs. The study demonstrates that the IQR-SIM can be applied across diverse simulation centers and program types, enhancing its broad use for implementation evaluation.
Related Concept Videos
01:19Quality Assurance
01:29Reliability and Validity
01:29Measures of Intelligence
Validity refers to how well a test measures what it claims to measure. An intelligence test should accurately assess intelligence rather than another characteristic, like anxiety. Criterion validity is one way to evaluate this;...
01:10Statgraphics
01:03Data Validation
Nursing assessment guides are generally based on holistic models rather than medical...
01:05Quality Control
Quality control helps track data, visualize trends, and identify variations, making it easier to detect deviations that may affect the accuracy of an analysis. One way to do this is by generating a quality control chart, which...