Related Experiment Video
Updated: Jun 16, 2026

Problem-Solving Before Instruction (PS-I): A Protocol for Assessment and Intervention in Students with Different Abilities
Published on: September 11, 2021
Reliability of a procedural checklist as a high-stakes measurement of advanced technical skill
Sarah E Peyré1, Christian G Peyré, Jeffrey A Hagen
1Department of Surgery, Brigham and Women's Hospital, 75 Francis Street, Neville House, Boston, MA 02115, USA. speyre@partners.org
Background:
The purpose of this study was to determine the reliability of a previously validated laparoscopic Nissen fundoplication procedural checklist as a measurement of advanced technical skill.
Methods:
Five surgeons, using a 65-step procedural checklist, independently evaluated 2 video recordings of expert surgeon operative performances of a laparoscopic Nissen fundoplication. Results were analyzed for percent agreement and Fleiss kappa correlation for each operation independently and combined as a whole.
Results:
Sixty-four of the 65 steps had 80% or higher percent agreement for both operative case A and B independently and when considered overall. The Fleiss kappa coefficients for operative case A (kappa = .95) and operative case B (kappa = .96) and overall (operative case A + B) (kappa = .95).
Conclusion:
The percentage agreement and kappa coefficients for all 3 analyses indicate a high degree of reliability (>.80) that would support the use of this instrument for high-stakes assessment of a laparoscopic Nissen fundoplication.
Related Concept Videos
Methods of Documentation III: PIE
Reliability and Validity
Measures of Intelligence
Validity refers to how well a test measures what it claims to measure. An intelligence test should accurately assess intelligence rather than another characteristic, like anxiety. Criterion validity is one way to evaluate this; it...
Quality Assurance
Data Validation
Key parameters for method validation include: