Related Experiment Video
Updated: May 19, 2026

Computerized Adaptive Testing System of Functional Assessment of Stroke
Published on: January 7, 2019
Objective structured assessment of technical skills and checklist scales reliability compared for high stakes
Anthony G Gallagher1, Gerald C O'Sullivan, Gerald Leonard
1School of Medicine, University College Cork, Cork, Ireland; National Surgical Training Centre, Royal College of Surgeons in Ireland, Dublin, Ireland.
Background:
The establishment of assessment reliability at the level of the individual trainee is an important attribute of assessment methodologies, particularly for doctors who have been failed. This issue is of particular importance for the process of competence assessment in the USA, UK, Australia and New Zealand.
Methods:
We use data from 19 applicants for higher surgical training in 2008 at the Royal College of Surgeons in Ireland to compare: (i) the objective structured assessment of technical skills (OSATS) method; and (ii) a procedure-specific checklist to assess surgical technical skills in the excision of a sebaceous cyst task by two experienced senior surgeons.
Results:
The overall interrater reliability (IRR) of the OSATS assessment as determined by a correlation coefficient was 0.507 (P < 0.03) and 0.67 with coefficient alpha, considerably below the accepted 0.8 level of IRR. The checklist's overall IRR was 0.89. Individually, only five (26%) of the OSATS assessments reached the 0.8 level of IRR in contrast to 18 (95%) of the checklist assessments.
Discussion:
We propose binary procedure-based assessment checklists as more reliable assessment instruments with more robust reproducibility.
Related Concept Videos
Measures of Intelligence
Validity refers to how well a test measures what it claims to measure. An intelligence test should accurately assess intelligence rather than another characteristic, like anxiety. Criterion validity is one way to evaluate this; it...
Reliability and Validity
Self-Report Tests of Personality
Self-Esteem
Multiple Comparison Tests
It would be easy to compare two samples using a significance alpha level of 0.05. In other words, there is only one sample pair to be compared. However, it would be difficult to identify a significantly different sample if the number...