Efficient segmentation of large-area skin images: a statistical evaluation

This paper is the second of a series describing research and development performed by the SPOTS project towards automating the process of monitoring pigmented lesions for change over time. Several parameters of the system are introduced along with the methods used for determining appropriate settings. We then describe our experimental setup and present a detailed statistical analysis to evaluate system performance.

There exist three sets of parameters in our system related to image segmentation, classification, and a region-growing step that can be individually tuned. The multiresolution hierarchical segmentation algorithm contains both a set of weights for the tristimulus color values used in calculating color differences and a contrast threshold used to identify root nodes that represent image segments. These were tuned using a synthetic image to simulate a pigmented lesion with irregular border and by estimating the parameters by successive approximations. The neural network classification used to detect lesion segments was trained using a 10-fold cross-validation, and the output classification threshold was set to yield a small false negatives ratio. Finally, there is a growing threshold used to determine whether neighbor regions should be merged in the region-growing step that was selected to minimize the change in area for different translations of a synthetic lesion. In order to evaluate the performance of the system, we compared the identified lesions with manually localized lesions. Ten images of the same area were taken from two subjects presenting lesions of varying size and different background skin texture. A sample of two images from the set was used to simulate variations in acquisition. Five operators were asked to trace lesion boundaries in two passes, separated by a few days, to evaluate the consistency of operators among themselves. The system was then run on the images using the optimal set of parameters. The results were analyzed for independent observation, matched pass, and doubly-matched pass data.

Independent observation data indicates that the system finds significantly more objects than any of the human operators, but the areas reported are not significantly different on average. Matched pass data show that the correlations of the system with human operators is quite high. The system agrees with humans only slightly less than they do among themselves and has a tendency to report slightly smaller areas. The correlation between system and human peformance was further reinforced by the doubly-matched arrangement. These results indicate that our system is as reliable and consistent as our human reference, yielding similar performance.