TRUST-TECH-Based Systematic Search for Multiple Local Optima in Deep Neural Nets

Training deep neural networks (DNNs) rested heavily on efficient local solvers. Due to their local property, local solvers are sensitive to initialization and hyperparameters. In this article, a systematical method for finding multiple high-quality local optimal DNNs, based on the transformation under stability-retaining equilibria characterization (TRUST-TECH) method, is introduced. Our goal is to systematically search for multiple local optimal parameters for large models, such as DNNs, trained on large datasets. To achieve this, a dynamic searching path (DSP) method is proposed to provide improved search guidance used in TRUST-TECH. By integrating the DSP method with the TRUST-TECH (DSP-TT) method, multiple optimal training solutions with higher quality than randomly initialized ones can be obtained. To take advantage of these optimal solutions, a DSP-TT ensemble method is further developed. Experiments on various test cases show that the proposed DSP-TT method achieves considerable improvement over other ensemble methods developed for deep architectures. The DSP-TT ensemble method also shows diversity advantages over other ensemble methods.