Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Entropy and the Second Law of Thermodynamics01:20

Entropy and the Second Law of Thermodynamics

2.9K
The second law of thermodynamics can be stated quantitatively using the concept of entropy. Entropy is the measure of disorder of the system.
The relation  between entropy and disorder can be illustrated with the example of the phase change of ice to water. In ice, the molecules are located at specific sites giving a solid state, whereas, in a liquid form, these molecules are much freer to move. The molecular arrangement has therefore become more randomized. Although the change in average...
2.9K
Entropy02:39

Entropy

30.4K
Salt particles that have dissolved in water never spontaneously come back together in solution to reform solid particles. Moreover, a gas that has expanded in a vacuum remains dispersed and never spontaneously reassembles. The unidirectional nature of these phenomena is the result of a thermodynamic state function called entropy (S). Entropy is the measure of the extent to which the energy is dispersed throughout a system, or in other words, it is proportional to the degree of disorder of a...
30.4K
Entropy Change in Reversible Processes01:10

Entropy Change in Reversible Processes

2.6K
In the Carnot engine, which achieves the maximum efficiency between two reservoirs of fixed temperatures, the total change in entropy is zero. The observation can be generalized by considering any reversible cyclic process consisting of many Carnot cycles. Thus, it can be stated that the total entropy change of any ideal reversible cycle is zero.
The statement can be further generalized to prove that entropy is a state function. Take a cyclic process between any two points on a p-V diagram.
2.6K
Convolution: Math, Graphics, and Discrete Signals01:24

Convolution: Math, Graphics, and Discrete Signals

305
In any LTI (Linear Time-Invariant) system, the convolution of two signals is denoted using a convolution operator, assuming all initial conditions are zero. The convolution integral can be divided into two parts: the zero-input or natural response and the zero-state or forced response, with t0 indicating the initial time.
To simplify the convolution integral, it is assumed that both the input signal and impulse response are zero for negative time values. The graphical convolution process...
305
Entropy within the Cell01:22

Entropy within the Cell

10.8K
A living cell's primary tasks of obtaining, transforming, and using energy to do work may seem simple. However, the second law of thermodynamics explains why these tasks are harder than they appear. None of the energy transfers in the universe are completely efficient. In every energy transfer, some amount of energy is lost in a form that is unusable. In most cases, this form is heat energy. Thermodynamically, heat energy is defined as the energy transferred from one system to another that...
10.8K
Convolution Properties II01:17

Convolution Properties II

239
The important convolution properties include width, area, differentiation, and integration properties.
The width property indicates that if the durations of input signals are T1 and T2, then the width of the output response equals the sum of both durations, irrespective of the shapes of the two functions. For instance, convolving two rectangular pulses with durations of 2 seconds and 1 second results in a function with a width of 3 seconds.
The area property asserts that the area under the...
239

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Automatic identification of chemical moieties.

Physical chemistry chemical physics : PCCP·2023
Same author

Long-Term Postoperative Pain Prediction Using Higher-Order Singular Value Decomposition of Intraoperative Physiological Responses: Prospective Cohort Study.

JMIR perioperative medicine·2022
Same author

Extracting synchronized neuronal activity from local field potentials based on a marked point process framework.

Journal of neural engineering·2022
Same author

Simple Stopping Criteria for Information Theoretic Feature Selection.

Entropy (Basel, Switzerland)·2020
Same author

Explaining decisions of deep neural networks used for fish age prediction.

PloS one·2020
Same journal

Research on a Regional Availability Evaluation Model for Road-Area High-Entropy Energy Based on Synergy Factors.

Entropy (Basel, Switzerland)·2026
Same journal

Atmospheric Turbulence Channel Modeling and Performance Analysis of a CO-ZP-OFDM Coherent Optical Communication System for UAV Air-to-Ground Scenarios.

Entropy (Basel, Switzerland)·2026
Same journal

Information Geometry and Asymptotic Theory for SMML Estimators.

Entropy (Basel, Switzerland)·2026
Same journal

Correlation Entropy and Power-Law Kinetics.

Entropy (Basel, Switzerland)·2026
Same journal

Research on the Contagion of Systemic Financial Risk Under the Impact of Climate Risks-From the Perspective of Complex Networks and Machine Learning.

Entropy (Basel, Switzerland)·2026
Same journal

The Statistical-Mechanical Meaning of the Wave Function of Quantum Mechanics.

Entropy (Basel, Switzerland)·2026
See all related articles

Related Experiment Video

Updated: Jul 25, 2025

Author Spotlight: Advancing Alzheimer's Research – Exploring Early Detection and Multi-Omics Approaches
09:47

Author Spotlight: Advancing Alzheimer's Research – Exploring Early Detection and Multi-Omics Approaches

Published on: December 15, 2023

1.2K

Analysis of Deep Convolutional Neural Networks Using Tensor Kernels and Matrix-Based Entropy.

Kristoffer K Wickstrøm1, Sigurd Løkse1, Michael C Kampffmeyer1,2

  • 1Machine Learning Group, Department of Physics and Technology, UiT The Arctic University of Norway, NO-9037 Tromsø, Norway.

Entropy (Basel, Switzerland)
|June 28, 2023
PubMed
Summary
This summary is machine-generated.

This study introduces a novel information plane (IP) analysis for deep neural networks (DNNs) using matrix-based Rényi

Keywords:
deep learninginformation planeinformation theorykernels methods

More Related Videos

Applications of EEG Neuroimaging Data: Event-related Potentials, Spectral Power, and Multiscale Entropy
11:15

Applications of EEG Neuroimaging Data: Event-related Potentials, Spectral Power, and Multiscale Entropy

Published on: June 27, 2013

33.8K
Author Spotlight: Enhancement of Salient Object Detection for Smart Grid Applications
03:31

Author Spotlight: Enhancement of Salient Object Detection for Smart Grid Applications

Published on: December 15, 2023

585

Related Experiment Videos

Last Updated: Jul 25, 2025

Author Spotlight: Advancing Alzheimer's Research – Exploring Early Detection and Multi-Omics Approaches
09:47

Author Spotlight: Advancing Alzheimer's Research – Exploring Early Detection and Multi-Omics Approaches

Published on: December 15, 2023

1.2K
Applications of EEG Neuroimaging Data: Event-related Potentials, Spectral Power, and Multiscale Entropy
11:15

Applications of EEG Neuroimaging Data: Event-related Potentials, Spectral Power, and Multiscale Entropy

Published on: June 27, 2013

33.8K
Author Spotlight: Enhancement of Salient Object Detection for Smart Grid Applications
03:31

Author Spotlight: Enhancement of Salient Object Detection for Smart Grid Applications

Published on: December 15, 2023

585

Area of Science:

  • Machine Learning
  • Artificial Intelligence
  • Information Theory

Background:

  • Information plane (IP) theory is a powerful tool for analyzing deep neural networks (DNNs), particularly their generalization capabilities.
  • Estimating mutual information (MI) for high-dimensional layers in DNNs and convolutional neural networks (CNNs) presents significant challenges for existing IP methods.
  • Current IP analysis techniques are limited and cannot be applied to large-scale CNNs.

Purpose of the Study:

  • To propose a novel and computationally tractable method for information plane analysis of deep neural networks, especially large-scale CNNs.
  • To overcome the limitations of existing MI estimators in handling high dimensionality and convolutional layers.
  • To provide new insights into the training dynamics and generalization abilities of large-scale neural networks.

Main Methods:

  • Developed a new information plane analysis approach utilizing matrix-based Rényi's entropy.
  • Integrated tensor kernels with Rényi's entropy to handle high-dimensional data and convolutional layers effectively.
  • Leveraged kernel methods for robust probability distribution representation, independent of data dimensionality.

Main Results:

  • Successfully applied the proposed method to perform a comprehensive IP analysis of large-scale CNNs.
  • Investigated different training phases of these networks, revealing new aspects of their behavior.
  • Provided novel insights into the training dynamics of large-scale neural networks, validating the approach on smaller DNNs.

Conclusions:

  • The proposed matrix-based Rényi's entropy and tensor kernel approach offers a scalable and robust solution for IP analysis of deep neural networks.
  • This method enables the study of complex, large-scale CNNs, overcoming limitations of previous techniques.
  • The findings offer a new perspective on understanding DNN generalization and training dynamics.