A Bayesian Approach to Recurrence in Neural Networks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Neuroscience

Background:

• Common neural network activation functions possess simple Bayesian origins.
• Bayes's theorem can imply a simple recurrence relation, forming the basis for novel network architectures.

Purpose of the Study:

• To develop a Bayesian recurrent unit with a prescribed feedback formulation.
• To investigate the impact of context indicators on feedback mechanisms.
• To create a probabilistic input gate within a Bayesian framework.

Main Methods:

• Derivation of a Bayesian recurrent unit based on Bayes's theorem.
• Introduction of a context indicator for variable feedback, analogous to forget mechanisms.
• Development of a probabilistic input gate using a similar Bayesian approach.
• Leveraging the forward-backward algorithm (Kalman smoother) for inference.

Main Results:

• The Bayesian recurrent unit naturally incorporates future and past inputs for inference.
• Experimental results on speech recognition demonstrate performance parity with bidirectional recurrent networks using fewer parameters.
• Explicitly bidirectional configurations of the Bayesian architecture outperform conventional bidirectional recurrent networks.

Conclusions:

• The proposed Bayesian recurrent unit offers a principled and effective alternative to conventional recurrent architectures.
• This approach provides a unified Bayesian perspective on recurrent neural network mechanisms.
• The architecture demonstrates strong performance in speech recognition tasks, highlighting its practical applicability.