The Work-ability Support Scale: evaluation of scoring accuracy and rater reliability

Area of Science:

• Rehabilitation Medicine
• Occupational Therapy
• Psychometrics

Background:

• Assessing vocational ability and support needs is crucial for individuals with acquired disabilities.
• The Work-ability Support Scale (WSS) was developed to aid decision-making in vocational rehabilitation.
• Evaluating the reliability and accuracy of new assessment tools is essential for their clinical utility.

Purpose of the Study:

• To report the iterative development and evaluation of the Work-ability Support Scale (WSS).
• To assess the inter- and intra-rater reliability and scoring accuracy of the WSS using vignettes.
• To highlight the impact of different methodological approaches on reliability analysis.

Main Methods:

• Six occupational therapists scored vignettes individually and in teams, with repeated blind scoring after one month.
• Scoring accuracy was tested against a reference standard using intraclass correlation coefficients (ICCs) and linear-weighted kappas (kw).
• Inter- and intra-rater reliability for individual and team scores were evaluated using two statistical methods.

Main Results:

• High scoring accuracy was observed (ICCs 0.78-0.96) for both parts of the WSS.
• Substantial to almost perfect agreement (kw ≥ 0.60) was found for most items, indicating good reliability.
• Team rating showed marginal improvement for Part A; four items required manual adjustment based on identified issues.

Conclusions:

• The vignette-based study indicates generally acceptable scoring accuracy and reliability for the WSS.
• The findings support the WSS as a viable tool for vocational rehabilitation assessment.
• Further validation of the WSS in real-life clinical settings is warranted.