Establishing Inter- and Intrarater Reliability for High-Stakes Testing Using Simulation

Suzan Kardong-Edgren1, Marilyn H Oermann, Mary Anne Rizzolo

  • 1About the Authors Suzan Kardong-Edgren, PhD, RN, CHSE, FAAN, ANEF, is a professor and director of the RISE Center, School of Nursing and Health Sciences, Robert Morris University, Moon Township, Pennsylvania. Marilyn H. Oermann, PhD, RN, FAAN, ANEF, is Thelma M. Ingles Professor of Nursing and director of evaluation and educational research, Duke University School of Nursing, Durham, North Carolina. Mary Anne Rizzolo, EdD, RN, FAAN, ANEF, is a consultant for the National League for Nursing. Tamara Odom-Maryon, PhD, is a professor of research, Washington State University College of Nursing, Spokane. For more information, contact Dr. Kardong-Edgren at kardongedgren@rmu.edu.

Summary

Developing standardized training for raters in high-stakes testing is crucial. This study found that not all faculty are expert evaluators, impacting reliability.

Related Concept Videos

Reliability and Validity01:29

Reliability and Validity

Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
14.2K
Modeling and Similitude01:12

Modeling and Similitude

Scaled modeling is a fundamental technique in engineering, enabling the study of large and complex systems by creating smaller, manageable replicas that recreate critical characteristics of the original. In hydrology and civil infrastructure, for example, scaled models of dams help analyze water flow, turbulence, and pressure. This method allows for accurate predictions of real-world behavior within a controlled environment, significantly reducing the cost and time involved in full-scale...
658
Bias01:22

Bias

Bias refers to any tendency that prevents a question from being considered unprejudiced. In research, bias occurs when one outcome or answer is selected or encouraged over others in sampling or testing. Bias can occur during any research phase, including study design, data collection, analysis, and publication.
In statistics, a sampling bias is created when a sample is collected from a population, and some members of the population are not as likely to be chosen as others (remember, each member...
7.4K
Accuracy and Errors in Hypothesis Testing01:13

Accuracy and Errors in Hypothesis Testing

Hypothesis testing is a fundamental statistical tool that begins with the assumption that the null hypothesis H0 is true. During this process, two types of errors can occur: Type I and Type II. A Type I error refers to the incorrect rejection of a true null hypothesis, while a Type II error involves the failure to reject a false null hypothesis.
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...
616
Uncertainty in Measurement: Accuracy and Precision03:37

Uncertainty in Measurement: Accuracy and Precision

Scientists typically make repeated measurements of a quantity to ensure the quality of their findings and to evaluate both the precision and the accuracy of their results. Measurements are said to be precise if they yield very similar results when repeated in the same manner. A measurement is considered accurate if it yields a result that is very close to the true or the accepted value. Precise values agree with each other; accurate values agree with a true value. 
107.7K