Interobserver variability in medical record review: an epidemiological study of asthma

Interobserver variability during the data collection for a large retrospective study of asthma was assessed. After extensive training by three physician investigators, two nurses collected study data from existing medical records using an explicit protocol. At four time intervals, samples of records already abstracted by one nurse were reviewed and re-abstracted by another nurse assigned to the project. The unweighted kappa coefficient was used to evaluate observer reliability. Most of the kappa coefficients were > or = 0.6, suggesting good to excellent agreement. Agreement was highest for demographic, clinical and medication variables, while judgement variables had the lowest kappa coefficients. The well known dependence of kappa on the prevalence of the variable studied was evident in these data. We strongly recommend periodic monitoring of observer reliability in any study using more than one individual to collect data to ensure reliable data collection.