Reverse-Engineering Neural Networks to Characterize Their Cost Functions.

Area of Science:

• Computational Neuroscience
• Machine Learning Theory

Background:

• Neural networks utilize cost functions to guide learning.
• Biologically plausible models require cost functions that align with neural activity and plasticity.

Purpose of the Study:

• To investigate biologically plausible cost functions in neural networks.
• To establish a formal link between neural network cost functions and variational free energy.
• To demonstrate how neural activity and plasticity relate to Bayesian inference and learning.

Main Methods:

• Formulating cost functions minimized by both neural activity and plasticity.
• Utilizing generative models based on partially observed Markov decision processes (POMDP).
• Applying mathematical and numerical analyses to establish theoretical equivalence.

Main Results:

• Cost functions are shown to be a variational bound on model evidence.
• Neural activity and plasticity perform Bayesian inference and learning, respectively, by maximizing model evidence.
• Formal equivalence is established between neural network cost functions and variational free energy under specific prior beliefs.

Conclusions:

• Neural network cost functions can be interpreted within a Bayesian inference framework.
• Optimal encoding of latent states is achieved when network priors match input-generating processes.
• This equivalence allows for hyperparameter optimization via variational free energy minimization and formal characterization of neural networks.