Related Experiment Videos
Online stabilization of block-diagonal recurrent neural networks
S C Sivakumar1, W Robertson, W J Phillips
1Department of Electrical Engineering, Dalhousie University-DalTech, Halifax, Canada.
IEEE Transactions on Neural Networks
|February 7, 2008
Summary
This study introduces the block-diagonal recurrent neural network (BDRNN) for simplified online training and enhanced stability. The BDRNN modifies backpropagation through time (BPTT) for reduced storage and direct stability monitoring during training.
Area of Science:
- Artificial Intelligence
- Machine Learning
- Computational Neuroscience
Background:
- Discrete-time recurrent neural networks (DTRNNs) present challenges in online training and stability.
- Conventional backpropagation through time (BPTT) can be computationally intensive and prone to instability.
Purpose of the Study:
- To introduce a novel Block-Diagonal Recurrent Neural Network (BDRNN) architecture.
- To develop a simplified online training approach for DTRNNs.
- To enhance network and training stability during learning.
Main Methods:
- Modified Backpropagation Through Time (BPTT) algorithm exploiting the BDRNN structure.
- Numerically stable recomputation of network state variables to reduce storage requirements.
- Development of a functional measure of system stability integrated into the cost function for direct stability monitoring.
Main Results:
- Demonstrated simplified online training capabilities of the BDRNN.
- Showcased effective reduction in storage requirements using the modified BPTT.
- Validated the direct monitoring and maintenance of network and training stability.
Conclusions:
- The BDRNN architecture offers a more efficient and stable approach to training recurrent neural networks.
- The proposed modifications to BPTT and the stability monitoring method are effective.
- BDRNNs provide a promising direction for stable and efficient recurrent neural network applications.
Related Concept Videos
01:37Neural Regulation
Digestion begins with a cephalic phase that prepares the digestive system to receive food. When our brain processes visual or olfactory information about food, it triggers impulses in the cranial nerves innervating the salivary glands and stomach to prepare for food.
01:22Block Diagram Reduction
The process of deriving the transfer function of a control system often involves reducing its block diagram to a single block. This simplification can be achieved through a series of strategic operations, including relocating branch points and comparators. These operations preserve the overall function of the system while allowing for easier manipulation and combination of blocks.
The first step in this process is the identification and relocation of a branch point. A branch point, where a...
The first step in this process is the identification and relocation of a branch point. A branch point, where a...
01:26Linearization and Approximation
Linearization is a mathematical technique used to approximate complex, nonlinear functions with simpler linear models in the vicinity of a chosen reference point. The method is based on the idea that, although a function may be difficult to evaluate exactly, its behavior near a specific input value can often be closely approximated by the tangent line at that point. This approach is particularly useful when small deviations from a known value are involved.Consider the square root function, for...
01:24Sequence Networks of Rotating Machines
A Y-connected synchronous generator, grounded through a neutral impedance, is designed to produce balanced internal phase voltages with only positive-sequence components. The generator's sequence networks include a source voltage that is exclusively in the positive-sequence network. The sequence components of line-to-ground voltages at the generator terminals illustrate this configuration.
Zero-sequence current induces a voltage drop across the generator's neutral impedance and other...
Zero-sequence current induces a voltage drop across the generator's neutral impedance and other...
01:22Multi-input and Multi-variable systems
Cruise control systems in cars are designed as multi-input systems to maintain a driver's desired speed while compensating for external disturbances such as changes in terrain. The block diagram for a cruise control system typically includes two main inputs: the desired speed set by the driver and any external disturbances, such as the incline of the road. By adjusting the engine throttle, the system maintains the vehicle's speed as close to the desired value as possible.
In the absence of...
In the absence of...
01:20Relation between Mathematical Equations and Block Diagrams
In a spring-mass-damper system, the second-order differential equation describes the dynamic behavior of the system. When transformed into the Laplace domain under zero initial conditions, this equation can be effectively analyzed and manipulated. The transformation into the Laplace domain converts differential equations into algebraic equations, simplifying the process of isolating the output.