Accuracy and reproducibility of visual coronary stenosis estimates using information from multiple observers

The reliability of visual estimation of severity of coronary artery stenosis may be improved using data from multiple independent observers. Data were collected from the results of a video format examination used on an experimental basis in 1987 by the American Board of Internal Medicine to test 61 candidates for certification in cardiovascular diseases. Twenty arteriographic cases were presented in a standardized format. Each artery was viewed in multiple projections including angled views. Each view was shown in both real time and slow motion, and each case was seen twice in its entirety. The observers rated stenosis severity on a four-point scale ranging from 1-4. A two-way repeated measures analysis of variance was performed on the tabulated results, yielding variance components for the arteriographic data (signal), the differences among observers, and the observer by case interaction (both considered noise). These components then allowed calculation of 68 and 95% confidence intervals, the signal-to-noise ratio, and the reproducibility coefficient for any number of observers. When a single observer was considered, reproducibility was low, with 95% confidence intervals of +/- 0.9 points, corresponding to approximately +/- 22% diameter stenosis. However, when data of three observers were averaged, the 95% confidence interval decreased to +/- 0.52 points (13% stenosis), signal-to-noise ratio rose to 12.2, and reproducibility coefficient was 0.92. Relatively small increments in these values were noted when data from a fourth or fifth observer were added. In comparison to a computer-assisted quantitative method, 86% accuracy was found for the results of averaged subjective determinations of stenosis severity.(ABSTRACT TRUNCATED AT 250 WORDS)