On-line node fault injection training algorithm for MLP networks: objective function and convergence analysis

Area of Science:

• Artificial Intelligence
• Machine Learning
• Neural Networks

Background:

• Fault tolerance in neural networks is a long-standing research area.
• Existing training algorithms for fault tolerance are numerous, but theoretical analysis remains incomplete.
• The on-line node fault injection algorithm, where hidden nodes randomly output zeros during training, lacks comprehensive theoretical understanding.

Purpose of the Study:

• To present the objective function for the on-line node fault injection algorithm.
• To provide a convergence proof for this algorithm applied to multilayer perceptrons (MLPs).
• To analyze the objective functions across different MLP output node configurations.

Main Methods:

• Theoretical analysis of the on-line node fault injection algorithm.
• Derivation of objective functions for MLPs with single linear, multiple linear, and single sigmoid output nodes.
• Convergence proof using mathematical analysis, showing convergence with probability one.

Main Results:

• The algorithm is shown to converge with probability one for MLPs.
• Objective functions for MLPs with single and multiple linear output nodes share a common form, including mean square errors, a regularizer, and weight decay terms.
• The objective function for MLPs with a single sigmoid output node differs slightly from the linear output cases.

Conclusions:

• The derived objective functions enable comparison of similarities and differences across various fault tolerance algorithms and MLP configurations.
• This work contributes to a more complete theoretical understanding of node fault injection-based training algorithms.
• The findings facilitate further research into robust neural network design and training methodologies.