Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Videos

Variational learning and bits-back coding: an information-theoretic view to Bayesian learning.

Antti Honkela1, Harri Valpola

  • 1Neural Networks Research Centre, Helsinki University of Technology, FI-02015 HUT, Finland. antti.honkela@hut.fi

IEEE Transactions on Neural Networks
|October 6, 2004
PubMed
Summary
This summary is machine-generated.

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

A supervised Bayesian method for time (re)annotation of transcriptomics data.

NAR genomics and bioinformatics·2026
Same author

Pan-pathogen deep sequencing of nosocomial bacterial pathogens in Italy in spring 2020: a prospective cohort study.

The Lancet. Microbe·2024
Same author

Collaborative learning from distributed data with differentially private synthetic data.

BMC medical informatics and decision making·2024
Same author

Digital public health leadership in the global fight for health security.

BMJ global health·2023
Same author

Strong pathogen competition in neonatal gut colonisation.

Nature communications·2022
Same author

Bacterial genomic epidemiology with mixed samples.

Microbial genomics·2021
Same journal

Universal perceptron and DNA-like learning algorithm for binary neural networks: LSBF and PBF implementations.

IEEE transactions on neural networks·2013
Same journal

Guest editorial: special section on white box nonlinear prediction models.

IEEE transactions on neural networks·2011
Same journal

Data-based fault-tolerant control of high-speed trains with traction/braking notch nonlinearities and actuator failures.

IEEE transactions on neural networks·2011
Same journal

Guest editorial: special section on data-based control, modeling, and optimization.

IEEE transactions on neural networks·2011
Same journal

Neural network-based multiple robot simultaneous localization and mapping.

IEEE transactions on neural networks·2011
Same journal

Data-driven model-free adaptive control for a class of MIMO nonlinear discrete-time systems.

IEEE transactions on neural networks·2011
See all related articles

Bits-back coding links Bayesian and minimum description length (MDL) learning. This approach offers new insights into variational Bayesian learning, model comparison, and pruning for hierarchical latent variable models.

Area of Science:

  • Machine Learning
  • Information Theory
  • Computational Neuroscience

Background:

  • Bits-back coding, introduced by Wallace (1990) and Hinton & van Camp (1993), connects Bayesian learning with minimum description length (MDL) principles.
  • Variational Bayesian methods, particularly ensemble learning, utilize cost functions that can be interpreted through the lens of information theory.
  • Understanding the relationship between Bayesian inference and information-theoretic approaches is crucial for advancing machine learning models.

Purpose of the Study:

  • To demonstrate the benefits of integrating Bayesian and information-theoretic viewpoints using bits-back coding.
  • To interpret the cost function in variational Bayesian ensemble learning as a code length.
  • To provide novel insights into the learning process and model components in hierarchical latent variable models.

Related Experiment Videos

Main Methods:

  • Utilizing bits-back coding to link Bayesian and MDL learning frameworks.
  • Applying variational Bayesian inference to hierarchical latent variable models.
  • Analyzing the cost function as a code length to understand posterior approximation misfit and model evidence bounds.

Main Results:

  • The bits-back coding framework provides a dual interpretation: Bayesian misfit and information-theoretic code length.
  • This dual view offers valuable insights into model comparison, pruning, and other aspects of the learning process.
  • The approach elucidates phenomena observed during the learning of hierarchical latent variable models.

Conclusions:

  • The integration of Bayesian and MDL perspectives via bits-back coding enhances the understanding of learning mechanisms.
  • This unified view is particularly beneficial for analyzing and optimizing hierarchical latent variable models.
  • The code-length interpretation offers a powerful tool for model selection and understanding model complexity in machine learning.