Learning multiple causes by competition enhanced least mean square error reconstruction

In this paper we studied a self-organization principle that input should be best reconstructed from a factorial distributed hidden representation, which has been addressed in the literature recently. An auto-encoder network is trained by the Least Mean Square Error Reconstruction (LMSER) while the redundance in the representation is reduced by a proposed anti-Hebbian scheme, in which a penalty term called Receptive Field Overlapping Index (RFOI) is combined into the objective function for enhancing competition among nodes in the network. Our learning scheme provides a way for balancing the cooperation and competition necessary for the self-organization process thus realizes the multiple causes model, which accounts for an observed data by combining assertions from the discovered causes or features in the data. Our experiment results demonstrate again the powerful information processing capability inherent to the popular weighted sum followed by sigmoid squashing. Comparing with previous probability theory based multiple causes models, our scheme is much easier to implement and quite reliable.