Inter-rater reliability in disability assessment based on a semi-structured interview report

To investigate whether physicians are able to perform a disability assessment based on a written report and to assess the inter-rater reliability between physicians in the assessment of work limitations.

In total, 12 insurance physicians used written reports to assess work limitations in 12 patients. The reports involved a semi-structured interview executed by a nurse practitioner. The insurance physicians were asked whether they could make reliable assessments based on these reports. In addition, inter-rater reliability was measured by computing their percentage agreement with respect to the mental and physical items of two Dutch disability lists (the Functional Information System and the Mental Ability List).

The quality of the reports was evaluated as reasonable to good. Half the physicians found the assessment based on the reports to be reasonably reliable, 25% found the opposite and 25% was indecisive. The overall agreement between the insurance physicians was reasonable to good, with a mean agreement of 76% (range 64 - 88%). Agreement between the physicians concerning the number of hours a patient could function daily was low.

Half the physicians thought that a reliable assessment based on the written information was possible. The quality of written patient reports made by nurse practitioners trained in conducting a semi-structured interview was considered reasonable to good by insurance physicians. The inter-rater reliability between insurance physicians of physical-disability and mental-disability assessment based on the written reports was reasonable to good. The assessment of the number hours patients could function daily had low inter-rater reliability.