Examining the Validity of the Implementation Quality Rubric for Simulation (IQR-SIM) for Assessing Implementation...

Area of Science:

• Healthcare Simulation
• Implementation Science
• Program Evaluation

Background:

• Reliable tools are needed to measure implementation processes and outcomes in healthcare simulation.
• Existing tools often lack validity, weakening confidence in study findings and strategy effectiveness.
• The Implementation Quality Rubric for Simulation (IQR-SIM) was developed to address this gap.

Purpose of the Study:

• To gather validity evidence for the Implementation Quality Rubric for Simulation (IQR-SIM).
• To assess the response process and internal structure of the IQR-SIM.
• To support the use of IQR-SIM in evaluating simulation-based programs.

Main Methods:

• The IQR-SIM, with a 3-point rating scale, was used to evaluate implementation data from semistructured interviews.
• Data were collected from 20 simulation-based programs across 7 international simulation centers.
• Inter-rater reliability (ICC=0.57) and internal consistency (Cronbach alpha=0.85) were assessed.

Main Results:

• The IQR-SIM demonstrated good internal consistency reliability (Cronbach alpha=0.85).
• Moderate inter-rater reliability (ICC=0.57) was observed, with values ranging from 0.39 to 0.52.
• Phase 1 of implementation (stakeholder engagement) showed the highest proportion of fully accomplished items.

Conclusions:

• Validity evidence for the IQR-SIM's response process and internal structure was established.
• The IQR-SIM is suitable for evaluating the implementation quality of simulation-based programs.
• The tool's applicability across diverse simulation centers supports its broad use in implementation evaluation.