Statistical nuclear texture analysis in cancer research: a review of methods and applications

In digital pathology, the field of nuclear texture analysis gives information about the spatial arrangement of the pixel gray levels in a digitized microscopic nuclear image, providing statistical texture measures that may be used as quantitative tools for diagnosis and prognosis of human cancer. In the present work, we have reviewed nuclear texture analysis in human cancer research, with emphasis on (i) statistical texture analysis methods, (ii) methods for feature evaluation and feature set selection, (iii) classification methods and error estimation, and (iv) the recent literature in the field, focusing on diagnosis- and prognosis-related applications. The application study covers the period from 1995 to 2007. In order to find nuclear features that discriminate robustly between cases from different diagnostic or prognostic classes, a statistical evaluation of features must be performed, and this demands careful experimental design. The present review reveals that it is quite common to evaluate a large number of features on a limited learning set of clinical material, without testing the chosen classifier on an independent validation data set. This easily leads to overoptimistic results. Out of 160 papers, we found only 30 papers in which the classifier was evaluated on an independent validation data set. Even in these studies, some good results have been hampered by small validation groups. However, it is encouraging to note that those publications meeting the requirements of an optimal study are generally showing good results. Thus, it is well documented that nuclear texture analysis is showing promising results as a novel diagnostic and/or prognostic marker. Hopefully, we will soon see that these promising studies will be replicated in large, prospective, multicenter trials.