Observer variation in an audit of charts of patients with rheumatoid arthritis

Arthritis Society physiotherapists (PTs) in Ontario adopted Problem Oriented Records (POR) to monitor patient care; goals were to encourage patient specific treatment plans and standardized records. After nine months in practice, the POR system was evaluated by an audit on charts of patients with Rheumatoid Arthritis. Audit teams of two PTs were created within each of five geographic areas. Auditors attended an audit instruction workshop. Two charts, selected randomly from the discharged patients of each of 38 therapists, were independently scored on 56 items by an audit team. The extent of agreement within the auditor team (observer variation) was measured, and adequacy scores computed for each chart. Adequate reliability of the audit instrument was established before examination of adequacy scores and making inferences about quality of care. The auditor agreement measures (kappa) varied from 0.13 to 0.97, mean = 0.74 and SD = 0.16; these estimates were indicative of adequate reliability of the audit form, although agreement scores were different amongst the five areas (P2 less than .01). POR adequacy scores varied from 10.0 to 93.1, mean = 64.5 and SD = 16.9. There were no differences in adequacy scores amongst the five areas (P2 = .61). To reduce observer variation, increased pre-audit emphasis is needed on auditor guidelines and training for items relating to assessment findings, problem identification, and treatment planning.