Operational definitions for the NINDS-AIREN criteria for vascular dementia: an interobserver study

Vascular dementia (VaD) is thought to be the most common cause of dementia after Alzheimer's disease. The commonly used International Workshop of the National Institute of Neurological Disorders and Stroke (NINDS) and the Association Internationale pour la Recherche et l'Enseignement en Neurosciences (AIREN) criteria for VaD necessitate evidence of vascular disease on CT or MRI of the brain. The purposes of our study were to operationalize the radiological part of the NINDS-AIREN criteria and to assess the effect of this operationalization on interobserver agreement.

Six experienced and 4 inexperienced observers rated a set of 40 MRI studies of patients with clinically suspected VaD twice using the NINDS-AIREN set of radiological criteria. After the first reading session, operational definitions were conceived, which were subsequently used in the second reading session. Interobserver reproducibility was measured by Cohen's kappa.

Overall agreement at the first reading session was poor (kappa=0.29) and improved slightly after application of the additional definitions (kappa=0.38). Raters in the experienced group improved their agreement from almost moderate (kappa=0.39) to good (0.62). The inexperienced group started out with poor agreement (kappa=0.17) and did not improve (kappa=0.18). The experienced group improved in both the large- and small-vessel categories, whereas the inexperienced group improved generally in the extensive white matter hyperintensities categories.

Considerable interobserver variability exists for the assessment of the radiological part of the NINDS-AIREN criteria. Use of operational definitions improves agreement but only for already experienced observers.