Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Videos

Discovering Neural Nets with Low Kolmogorov Complexity and High Generalization Capability.

Jurgen Schmidhuber1

  • 1IDSIA, Switzerland

Neural Networks : the Official Journal of the International Neural Network Society
|July 1, 1997
PubMed
Summary

This study introduces a novel method using algorithmic complexity and a universal prior to find simple neural networks. This approach enhances generalization capabilities beyond traditional algorithms, particularly for specific problems.

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Accelerating the inference of string generation-based chemical reaction models for industrial applications.

Journal of cheminformatics·2025
Same author

emg2vec: Self-supervised Pretraining in Electromyography-based Silent Speech Interfaces.

Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual International Conference·2025
Same author

On the Distillation of Stories for Transferring Narrative Arcs in Collections of Independent Media.

IEEE transactions on pattern analysis and machine intelligence·2024
Same author

Reagent prediction with a molecular transformer improves reaction data quality.

Chemical science·2023
Same author

Unsupervised Learning of Temporal Abstractions With Slot-Based Transformers.

Neural computation·2023
Same author

Recurrent Neural-Linear Posterior Sampling for Nonstationary Contextual Bandits.

Neural computation·2022

Area of Science:

  • Machine Learning
  • Algorithmic Information Theory
  • Neural Networks

Background:

  • Current neural network learning algorithms often seek simplicity for better generalization, adhering to Occam's razor.
  • Existing simplicity measures lack the universality and power of Kolmogorov complexity and Solomonoff's algorithmic probability.
  • Many Bayesian approaches struggle with selecting appropriate prior distributions.

Purpose of the Study:

  • To address limitations in simplicity measures and prior selection for neural network learning.
  • To introduce a probabilistic method for finding algorithmically simple solutions with high generalization capability.

Main Methods:

  • Review of algorithmic complexity theory and the Solomonoff-Levin universal prior.
  • Development of a probabilistic search method based on Levin complexity and optimal universal search.

Related Experiment Videos

  • Use of efficient, self-sizing programs to generate solution candidates influencing runtime and storage.

    • Main Results:

    • Demonstration of a method to discover neural networks with low Kolmogorov complexity and high generalization.
    • Simulations on toy problems show superior generalization results compared to previous neural network algorithms.
    • The method effectively identifies 'good' programs that compute algorithmically probable solutions fitting training data.

    Conclusions:

    • The proposed method, grounded in algorithmic probability, offers a powerful alternative for neural network learning.
    • Significant improvements in generalization are achievable, outperforming existing algorithms on specific tasks.
    • Further research is needed for large-scale applications and incremental learning.