Entropy Estimation: From Statistical Physics to Classification

Driven by the common entropy formalism in statistical physics and information theory, here we show that methods developed for the analysis of molecular configurational ensembles may be successfully used for classification. Entropy is estimated using the k-th nearest neighbor method combined with the Maximum Information Spanning Tree method to account for the mutual information between variables. This formalism is used to classify new samples using their estimated cross-entropy with the training set. The method is tested using the hand-written digits MNIST and the functional genomics CuMiDa datasets. The results obtained on these diverse datasets are comparable to those obtained by machine learning methods, or better when the number of samples is small. The methods described here have the advantage, compared to other classification methods, of directly linking variables and pairs of variables to the assigned class, a valuable feature when aiming at identifying causal relationships between features and classes (e.g., the relationship between genes and diseases).