Using Generalizability Theory to Inform Optimal Design for a Nursing Performance Assessment

Area of Science:

• Healthcare Education
• Nursing Practice
• Psychometrics

Background:

• Assessing healthcare professional competency is crucial.
• Simulation-based assessments are valuable for clinical skills but often lack rigorous psychometric evaluation.
• Reliability in these assessments needs careful consideration beyond simple rater agreement.

Purpose of the Study:

• To examine the psychometric properties of the Nursing Performance Profile using generalizability (G) theory.
• To illustrate the application of G theory in analyzing error variance in simulation-based assessments.
• To provide recommendations for designing and evaluating simulation-based assessments in healthcare.

Main Methods:

• Utilized extant data from 18 nurses assessed on 3 scenarios by 3 raters.
• Applied generalizability (G) theory to analyze the psychometric characteristics of the Nursing Performance Profile.
• Examined sources of measurement error, including scenarios, raters, and items.

Main Results:

• Findings indicate that achieving desired score reliability necessitates a significantly larger number of scenarios and/or raters.
• Demonstrated how G theory can quantify the relative impact of different error sources and their interactions.
• Confirmed previous research on the challenges of reliability in simulation-based assessments.

Conclusions:

• Simulation-based competency assessments require careful design to ensure reliability and validity.
• Generalizability theory is a powerful tool for understanding and improving the psychometric quality of performance assessments.
• Recommendations are provided for future research and development of healthcare simulation assessments.