Related Experiment Videos
American Occupational Therapy Association Accreditation Committee Reliability Study
Summary
This study assessed the reliability of accreditation raters from the Accreditation Council for Continuing Medical Education (ACCME), Review and Action (RAE), and Accreditation and Maintenance (AMA) programs. Results show high internal consistency and interrater reliability, ensuring consistent evaluation of accreditation Essentials.
Area of Science:
- Accreditation science
- Quality assurance in education
- Professional standards evaluation
Background:
- Previous validity and reliability studies by CAHEA and AOTA Commission on Education.
- Need for a reliability study on accrediting activity processes within AOTA and CAHEA.
Purpose of the Study:
- Augment existing validity and reliability studies.
- Establish internal consistency of AC, RAE, and AMA raters.
- Determine interrater reliability and accuracy among AC, RAE, and AMA raters.
Main Methods:
- Assessed internal consistency for AC, RAE, and AMA raters.
- Calculated interrater reliability among all raters and specific subgroups (AC, RAE).
- Analyzed rater accuracy at different decision points, including the Report of On-Site Evaluation (ROSE).
Main Results:
- RAE raters demonstrated highest internal consistency (.94), followed by AC (.93) and AMA (.88).
- Overall interrater reliability was high (.93), with AC and RAE raters achieving .95 agreement.
- Rater concordance increased with more data, reaching .83 to .96 agreement by Decision 4 (ROSE).
Conclusions:
- Accreditation raters exhibit high internal consistency and interrater reliability.
- The findings support consistent and dependable evaluation of accreditation Essentials.
- Recent AOTA accreditation orientation may contribute to improved rater consistency.