Predicting the Future: Advantages of Semilocal Units

Area of Science:

• Artificial Intelligence
• Machine Learning
• Neural Networks

Background:

• Radial Basis Function (RBF) networks excel in low-dimensional problems but struggle with high-dimensional input spaces due to their localized nature.
• Standard sigmoid unit backpropagation is slower for low-dimensional problems but offers advantages in high-dimensional spaces.

Purpose of the Study:

• To develop a novel activation function that combines the speed of RBFs with the nonlocal properties of sigmoids.
• To improve the accuracy and learning speed of neural networks for nonlinear time series modeling, especially in the presence of irrelevant input variables.

Main Methods:

• Introduction of a new class of semilocal activation functions, termed 'gaussian bar' units.
• Comparative analysis of gaussian bar networks against RBF and sigmoid networks on Mackey-Glass and Coupled Lattice Map problems.
• Investigation of 'composed gaussians' (gaussians-of-gaussians) within gaussian bar architectures.
• Exploration of 'automatic connection pruning' in gaussian bar functions under gradient descent.

Main Results:

• Gaussian bar networks demonstrate superior accuracy compared to RBF networks when irrelevant inputs are present.
• Gaussian bar networks achieve significant speedups over sigmoid networks, comparable to RBF network training times.
• Networks utilizing composed gaussians within the gaussian bar architecture exhibit the highest performance.
• Automatic connection pruning is identified as a key factor in the success of gaussian bar representations.

Conclusions:

• Gaussian bar units provide an effective solution for modeling nonlinear time series, overcoming limitations of both RBF and sigmoid networks.
• The proposed semilocal activation functions offer a robust and efficient alternative for complex time series analysis.
• Composed gaussians and automatic connection pruning are crucial elements for achieving optimal performance with gaussian bar networks.