Premature Saturation in Backpropagation Networks: Mechanism and Necessary Conditions

The mechanism that gives rise to the phenomenon of premature saturation of the output units of feedforward multilayer neural networks during training with the standard backpropagation algorithm is described. The entire process of premature saturation is characterized by three distinct stages, and it is concluded that the momentum term plays the leading role in the occurrence of the phenomenon. The necessary conditions for the occurrence of premature saturation are presented, and a method is proposed, based on these conditions, that eliminates the occurrence of the phenomenon. The validity of the conditions and the proposed method are illustrated through simulation results. Copyright 1997 Elsevier Science Ltd.