ROC analysis of observer-response subjective rating data--application to periodontal radiograph assessment

Many physical anthropological studies require that an observer or device discriminate between states that can be easily confused. Receiver operating characteristic (ROC) analysis currently offers the best method for determining the accuracy of such choices, particularly for small sample sizes. Although ROC analysis is widely accepted in psychophysical and biomedical testing, its use in anthropological studies has not been reported. ROC analysis is used here to determine the usefulness of enhanced dental radiographs to assess vertical alveolar bone defects for quantitative studies of human variation with regard to periodontal disease. The presence or absence of vertical-bony defects (truth) for 75 human skulls was established by the consensus of two trained observers. Dental bitewing-radiographs were taken of the alveolar processes, the radiographs digitized, and the brightness and contrast of the digital images enhanced. The two observers who established truth then rated 1) plain bitewing radiographs, 2) unenhanced digital images of bitewings, and 3) enhanced digital images of bitewings for vertical bony defects. The rating scale varied from 1 (vertical defect definitely or almost definitely present) to 5 (definitely or almost definitely absent). ROC analysis was used to compared the diagnostic value of the 3 imaging modalities. All modalities had nearly identical diagnostic performance, measured as Az values (areas beneath ROC curves) that were less than 0.80, which indicates only moderate usefulness. It is concluded that enhancement does not increase success in vertical-bony-defect diagnosis from digital dental radiographs processed in this manner. Moreover, it is suggested that conventional bitewing radiographs may be unsuitable for accurate quantification of such defects.