Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Neural Regulation01:37

Neural Regulation

42.2K
Digestion begins with a cephalic phase that prepares the digestive system to receive food. When our brain processes visual or olfactory information about food, it triggers impulses in the cranial nerves innervating the salivary glands and stomach to prepare for food.
42.2K
Regression Toward the Mean01:52

Regression Toward the Mean

6.6K
Regression toward the mean (“RTM”) is a phenomenon in which extremely high or low values—for example, and individual’s blood pressure at a particular moment—appear closer to a group’s average upon remeasuring. Although this statistical peculiarity is the result of random error and chance, it has been problematic across various medical, scientific, financial and psychological applications. In particular, RTM, if not taken into account, can interfere when...
6.6K
Multi-input and Multi-variable systems01:22

Multi-input and Multi-variable systems

273
Cruise control systems in cars are designed as multi-input systems to maintain a driver's desired speed while compensating for external disturbances such as changes in terrain. The block diagram for a cruise control system typically includes two main inputs: the desired speed set by the driver and any external disturbances, such as the incline of the road. By adjusting the engine throttle, the system maintains the vehicle's speed as close to the desired value as possible.
In the absence of...
273
Region of Convergence01:17

Region of Convergence

735
The z-transform is a powerful mathematical tool used in the analysis of discrete-time signals and systems. It is a crucial tool in the analysis of discrete-time systems, but its convergence is limited to specific values of the complex variable z. This range of values, known as the Region of Convergence (ROC), is fundamental in determining the behavior and stability of a system or signal. The ROC defines the region in the complex plane where the z-transform converges, which can take various...
735
Propagation of Uncertainty from Random Error00:59

Propagation of Uncertainty from Random Error

1.5K
An experiment often consists of more than a single step. In this case, measurements at each step give rise to uncertainty. Because the measurements occur in successive steps, the uncertainty in one step necessarily contributes to that in the subsequent step. As we perform statistical analysis on these types of experiments, we must learn to account for the propagation of uncertainty from one step to the next. The propagation of uncertainty depends on the type of arithmetic operation performed on...
1.5K
Generalization, Discrimination, and Extinction01:24

Generalization, Discrimination, and Extinction

1.1K
Generalization, discrimination, and extinction are key concepts in operant conditioning that influence how behaviors are learned and maintained.
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
1.1K

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

High-performance deep spiking neural networks with 0.3 spikes per neuron.

Nature communications·2024
Same author

On the visual analytic intelligence of neural networks.

Nature communications·2023
Same author

Robust high-dimensional memory-augmented neural networks.

Nature communications·2021
See all related articles

Related Experiment Video

Updated: Nov 27, 2025

Deep Neural Networks for Image-Based Dietary Assessment
13:19

Deep Neural Networks for Image-Based Dietary Assessment

Published on: March 13, 2021

9.7K

Convergence Behavior of DNNs with Mutual-Information-Based Regularization.

Hlynur Jónsson1, Giovanni Cherubini1, Evangelos Eleftheriou1

  • 1IBM Research Zurich, 8803 Rüschlikon, Switzerland.

Entropy (Basel, Switzerland)
|December 8, 2020
PubMed
Summary

This study uses information theory to analyze Deep Neural Networks (DNNs), confirming input compression in complex Convolutional Neural Networks (CNNs). Regularizing DNNs with mutual information estimates stabilizes test accuracy and reduces variance.

Keywords:
deep neural networksinformation bottleneckregularization methods

More Related Videos

Author Spotlight: Enhancement of Salient Object Detection for Smart Grid Applications
03:31

Author Spotlight: Enhancement of Salient Object Detection for Smart Grid Applications

Published on: December 15, 2023

823
Quantification of Information Encoded by Gene Expression Levels During Lifespan Modulation Under Broad-range Dietary Restriction in C. elegans
09:23

Quantification of Information Encoded by Gene Expression Levels During Lifespan Modulation Under Broad-range Dietary Restriction in C. elegans

Published on: August 16, 2017

8.4K

Related Experiment Videos

Last Updated: Nov 27, 2025

Deep Neural Networks for Image-Based Dietary Assessment
13:19

Deep Neural Networks for Image-Based Dietary Assessment

Published on: March 13, 2021

9.7K
Author Spotlight: Enhancement of Salient Object Detection for Smart Grid Applications
03:31

Author Spotlight: Enhancement of Salient Object Detection for Smart Grid Applications

Published on: December 15, 2023

823
Quantification of Information Encoded by Gene Expression Levels During Lifespan Modulation Under Broad-range Dietary Restriction in C. elegans
09:23

Quantification of Information Encoded by Gene Expression Levels During Lifespan Modulation Under Broad-range Dietary Restriction in C. elegans

Published on: August 16, 2017

8.4K

Area of Science:

  • Artificial Intelligence
  • Information Theory
  • Machine Learning

Background:

  • Deep Neural Networks (DNNs) are complex models where understanding training dynamics is crucial.
  • The information plane visualizes mutual information between variables during training, offering insights into DNN behavior.
  • Estimating mutual information for high-dimensional data in complex DNNs remains a significant challenge.

Purpose of the Study:

  • To analyze the information plane dynamics of high-dimensional Convolutional Neural Networks (CNNs).
  • To confirm the existence of an input compression phase in complex DNNs.
  • To investigate the benefits of mutual information-based regularization for DNN training.

Main Methods:

  • Leveraging information theory concepts to analyze DNNs.
  • Utilizing Mutual Information Neural Estimation (MINE) for high-dimensional mutual information computation.
  • Applying MINE to a VGG-16 CNN model to visualize its information plane.
  • Incorporating mutual information estimates into the DNN loss function for regularization.

Main Results:

  • Demonstrated the convergence of mutual information on the information plane for a VGG-16 CNN.
  • Confirmed the input compression phase in complex, high-dimensional DNNs.
  • Showcased that mutual information-based regularization stabilizes test accuracy and reduces variance, especially with extensive training.

Conclusions:

  • Mutual Information Neural Estimation (MINE) enables the analysis of information plane dynamics in complex DNNs like CNNs.
  • Mutual information-based regularization is an effective technique to improve DNN training stability and performance.
  • This work extends previous findings from low-dimensional networks to high-dimensional CNNs, addressing an open problem in the field.