A new method of assessment of clinical teaching: ROC analysis

Twenty-five clinical dental students with varying clinical experience were presented with case records consisting of a clinical history and a panoramic radiograph for 25 patients with bilateral lower third molars ('wisdom teeth'). The students were asked to indicate how certain they were that each lower third molar tooth needed removal using a 6-point rating scale. Immediately following the task, the students were presented with information on the indications for removal of lower third molars in the form of a lecture by a senior academic clinical teacher. One week later the students were asked to repeat the rating study, under the same conditions as before, using a further 25 clinical cases. Receiver operating characteristics (ROC) analysis, which provides a graphical and quantitative assessment of a group of observers' ability to detect need for treatment, was utilized to examine differences between junior and senior students and between the matched pre-training and post-training experiments. The ability of junior students to assign lower third molars for surgery was statistically no better than random selection of cases. Formal clinical teaching significantly improved this group's performance, but had no effect on the performance of senior students. Senior students were significantly better able correctly to assign lower third molars for surgical intervention than junior students. Therefore this study shows that clinical experience has a significantly greater influence on treatment-planning ability than formal teaching. ROC analysis is a useful tool for assessing the effectiveness of methods of undergraduate training.