An instrument for quality assurance in work capacity evaluation: development, evaluation, and inter-rater reliability

Area of Science:

• Insurance Medicine
• Occupational Health
• Medical Quality Assurance

Background:

• Disability pensions require work capacity evaluations, but external quality assurance guidelines are lacking.
• The quality of medical reports in insurance medicine varies, with scarce systematic evaluations.
• Previous reliability studies using peer review have shown insufficient ability to distinguish report quality.

Purpose of the Study:

• To develop and evaluate an instrument for examining the quality of medical experts' reports in disability evaluations.
• To assess the reliability and inter-rater agreement of the developed peer review instrument.

Main Methods:

• A peer review manual with six quality domains and 22 items was developed, including a superordinate criterion (survey confirmability).
• The manual was pre-tested with experienced physicians.
• Twenty anonymous expert reports were reviewed by 19 trained raters to evaluate inter-rater reliability using percentage agreement and Kendall's W.

Main Results:

• Agreement for the superordinate criterion ranged from 29.2% to 87.5%.
• Kendall's W for quality domain items varied widely (0.09 to 0.88).
• Kendall's W for the superordinate criterion was 0.39, indicating fair agreement, with systemic peer preferences noted.

Conclusions:

• The superordinate criterion's reliability was insufficient but comparable to other studies.
• Further efforts are needed to improve evaluation instruments for insurance medicine reports.
• Revising the peer review instrument and implementing standardized rater training are proposed to enhance reliability.