The reliability and agreement of subtask assessments

Area of Science:

• Educational Psychology
• Motor Learning
• Skill Acquisition

Background:

• Knowledge of results (KR) provides essential error feedback for motor skill learning.
• Inconsistent KR from different instructors can impede students' error correction and skill development.
• Reliable and consistent feedback is vital for effective student learning in skill-based education.

Purpose of the Study:

• To evaluate the effectiveness of a training program aimed at enhancing interrater agreement and assessment reliability for motor subtasks.
• To determine if targeted training can improve the consistency of knowledge of results (KR) provided to students.
• To investigate the impact of improved interrater agreement on the quality of feedback in skill-based training.

Main Methods:

• A training program involving case study discussions and group grading was implemented.
• Participants engaged in the group grading of prepared teeth as a practical application.
• Pre- and post-training data were collected and compared to assess changes in agreement and reliability.

Main Results:

• The training program led to a significant improvement in interrater agreement for assessing motor subtasks.
• Post-training assessments demonstrated enhanced reliability in the feedback provided by instructors.
• Data indicated a reduction in variability of assessments after the intervention.

Conclusions:

• Interrater agreement and assessment reliability for motor subtasks can be significantly improved through targeted training.
• Consistent KR, facilitated by improved instructor agreement, benefits students by providing more dependable error information.
• The findings support the implementation of such training programs to enhance the quality of feedback in skill-based educational settings.