Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Video

Updated: Jun 27, 2026

DeepOmicsAE: Representing Signaling Modules in Alzheimer's Disease with Deep Learning Analysis of Proteomics, Metabolomics, and Clinical Data
09:47

DeepOmicsAE: Representing Signaling Modules in Alzheimer's Disease with Deep Learning Analysis of Proteomics, Metabolomics, and Clinical Data

Published on: December 15, 2023

Algorithmic Compression via Pretrained Neural Networks.

Tim Genewein1, Jordi Grau-Moya1, Li Kevin Wenliang1

  • 1Google DeepMind, London N1C 4DJ, UK.

Entropy (Basel, Switzerland)
|June 26, 2026
PubMed
Summary
This summary is machine-generated.

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Simplicity and Complexity in Combinatorial Optimization.

Entropy (Basel, Switzerland)·2026
Same author

Modeling the Arrows of Time with Causal Multibaker Maps.

Entropy (Basel, Switzerland)·2024
Same author

Deep Neural Networks for Modeling Visual Perceptual Learning.

The Journal of neuroscience : the official journal of the Society for Neuroscience·2018
Same journal

Research on a Regional Availability Evaluation Model for Road-Area High-Entropy Energy Based on Synergy Factors.

Entropy (Basel, Switzerland)·2026
Same journal

Atmospheric Turbulence Channel Modeling and Performance Analysis of a CO-ZP-OFDM Coherent Optical Communication System for UAV Air-to-Ground Scenarios.

Entropy (Basel, Switzerland)·2026
Same journal

Information Geometry and Asymptotic Theory for SMML Estimators.

Entropy (Basel, Switzerland)·2026
Same journal

Correlation Entropy and Power-Law Kinetics.

Entropy (Basel, Switzerland)·2026
Same journal

Research on the Contagion of Systemic Financial Risk Under the Impact of Climate Risks-From the Perspective of Complex Networks and Machine Learning.

Entropy (Basel, Switzerland)·2026
Same journal

The Statistical-Mechanical Meaning of the Wave Function of Quantum Mechanics.

Entropy (Basel, Switzerland)·2026
See all related articles

Large neural networks trained for next-token prediction implicitly perform algorithmic compression, approximating Bayesian inference. This offers a theoretical framework for understanding their advanced reasoning and problem-solving capabilities.

Area of Science:

  • Artificial Intelligence
  • Machine Learning Theory
  • Algorithmic Information Theory

Background:

  • Large neural networks trained on vast datasets exhibit complex reasoning, resembling planning and search.
  • This emergent behavior challenges traditional computational and intelligence theories.
  • Bridging the gap between practical Large Language Models (LLMs) and formal theories is crucial.

Purpose of the Study:

  • To review theoretical and empirical work connecting LLM success to formal computation and intelligence theories.
  • To propose a framework where next-token prediction meta-trains models for algorithmic compression and Bayesian inference.
  • To explore the implications of algorithmic information theory for understanding modern AI.

Main Methods:

  • Grounded in memory-based meta-learning, the study analyzes how sequence models perform implicit meta-training.
Keywords:
Bayesian inferencealgorithmic inductioncompression and LLMslossless compressionmeta-learning

Related Experiment Videos

Last Updated: Jun 27, 2026

DeepOmicsAE: Representing Signaling Modules in Alzheimer's Disease with Deep Learning Analysis of Proteomics, Metabolomics, and Clinical Data
09:47

DeepOmicsAE: Representing Signaling Modules in Alzheimer's Disease with Deep Learning Analysis of Proteomics, Metabolomics, and Clinical Data

Published on: December 15, 2023

  • The core argument posits that predicting the next token across diverse tasks leads to algorithmic compression.
  • Theoretical and empirical evidence is reviewed to support the connection to Solomonoff induction and Bayesian inference.

    • Main Results:

    • Pretrained neural networks infer generative algorithms from data, effectively compressing information.
    • The approach approximates Solomonoff induction and matches exact Bayesian inference on complex data.
    • Models demonstrate strong compression on out-of-distribution data and synthesize complex algorithms like chessboard evaluations.

    Conclusions:

    • Understanding LLMs through algorithmic information theory provides insights into their capabilities and limitations.
    • As models advance, theoretical grounding in algorithmic information theory becomes increasingly vital.
    • Open research questions are outlined to connect theoretical understanding with practical machine learning advancements.