Related Experiment Video
Updated: Dec 17, 2025

Creation of a High-Fidelity, Low-Cost, Intraosseous Line Placement Task Trainer via 3D Printing
Published on: August 17, 2022
The Effect of Evaluator Training on Inter- and Intrarater Reliability in High-Stakes Assessment in Simulation
Ann E Holland1, Jone Tiffany, Linda Blazovich
1About the Authors Ann E. Holland, PhD, RN, is a professor, Bethel University, St. Paul, Minnesota. Jone Tiffany, DNP, RN, CNE, CHSE, ANEF, is a professor, Bethel University. Linda Blazovich, DNP, RN, CNE, is an associate professor, St. Catherine University, St. Paul, Minnesota. Deborah Bambini, PhD, RN, WHNP-BC, CNE, CHSE, ANEF, is a professor, Grand Valley State University, Grand Rapids, Michigan. Vicki Schug, PhD, RN, CNE, is a professor, St. Catherine University. This study was supported by the National League for Nursing, with funding from Laerdal Medical. The authors acknowledge the work of their research assistant, Dorie Fritz, MSN, RN, and statistician, Joel Frederickson, PhD. For more information, contact Dr. Holland at ann-holland@bethel.edu.
Aim:
The aim of this study was to evaluate the effectiveness of a training intervention in achieving inter- and intrarater reliability among faculty raters conducting high-stakes assessment of clinical performance in simulation.
Background:
High-stakes assessment of simulation performance is being adopted in nursing education. However, limited research exists to guide best practices in training raters, which is essential to ensure fair and defensible assessment.
Method:
A nationwide sample of 75 prelicensure RN program faculty participated in an experimental, randomized, controlled study.
Results:
Participants completing a training intervention achieved higher inter- and intrarater reliability than control group participants when using a checklist evaluation tool. Mixed results were achieved by participants when completing a global competency assessment.
Conclusion:
The training intervention was effective in helping participants to achieve a shared mental model for use of a checklist, but more time may be necessary to achieve consistent global competence decisions.
Related Concept Videos
Reliability and Validity
Self-Evaluation: Self-Enhancement and Self-Verification
Sources of Self-Esteem II: Performance Feedback
Theory of Attribution II: Kelley's Covariation Theory

