Feature Selection based on the Local Lift Dependence Scale

This paper uses a classical approach to feature selection: minimization of a cost function applied on estimated joint distributions. However, in this new formulation, the optimization search space is extended. The original search space is the Boolean lattice of features sets (BLFS), while the extended one is a collection of Boolean lattices of ordered pairs (CBLOP), that is (features, associated value), indexed by the elements of the BLFS. In this approach, we may not only select the features that are most related to a variable Y, but also select the values of the features that most influence the variable or that are most prone to have a specific value of Y. A local formulation of Shannon's mutual information, which generalizes Shannon's original definition, is applied on a CBLOP to generate a multiple resolution scale for characterizing variable dependence, the Local Lift Dependence Scale (LLDS). The main contribution of this paper is to define and apply the LLDS to analyse local properties of joint distributions that are neglected by the classical Shannon's global measure in order to select features. This approach is applied to select features based on the dependence between: i-the performance of students on university entrance exams and on courses of their first semester in the university; ii-the congress representative party and his vote on different matters; iii-the cover type of terrains and several terrain properties.