Search research articles

ABOUT JoVE

Overview Leadership Blog JoVE Help Center

AUTHORS

Publishing Process Editorial Board Scope & Policies Peer Review FAQ Submit

LIBRARIANS

Testimonials Subscriptions Access Resources Library Advisory Board FAQ

RESEARCH

JoVE Journal Methods Collections JoVE Encyclopedia of Experiments Archive

EDUCATION

JoVE Core JoVE Business JoVE Science Education JoVE Lab Manual Faculty Resource Center Faculty Site

Terms & Conditions of Use

Related Concept Videos

Introduction to Learning

Introduction to Learning

Learning is the process of acquiring knowledge or skills through practice or experience, leading to long-lasting behavioral changes. This acquisition occurs through interaction with the environment and requires practice or experience. For instance, mastering a skill such as surfing requires considerable practice and experience, highlighting the essential role of repeated interactions with the environment in learning.
In contrast to learned behaviors, unlearned behaviors such as crying, sexual...

Entropy Change in Reversible Processes

Entropy Change in Reversible Processes

In the Carnot engine, which achieves the maximum efficiency between two reservoirs of fixed temperatures, the total change in entropy is zero. The observation can be generalized by considering any reversible cyclic process consisting of many Carnot cycles. Thus, it can be stated that the total entropy change of any ideal reversible cycle is zero.
The statement can be further generalized to prove that entropy is a state function. Take a cyclic process between any two points on a p-V diagram.

Associative Learning

Associative Learning

Associative learning is a fundamental concept in behavioral psychology, wherein a connection is established between two stimuli or events, leading to a learned response. This process is critical in understanding how behaviors are acquired and modified. Conditioning, the mechanism through which associations are formed, can be divided into two main types: classical conditioning and operant conditioning, each elucidating different aspects of associative learning.
Classical conditioning, also known...

Entropy and the Second Law of Thermodynamics

Entropy and the Second Law of Thermodynamics

The second law of thermodynamics can be stated quantitatively using the concept of entropy. Entropy is the measure of disorder of the system.
The relation between entropy and disorder can be illustrated with the example of the phase change of ice to water. In ice, the molecules are located at specific sites giving a solid state, whereas, in a liquid form, these molecules are much freer to move. The molecular arrangement has therefore become more randomized. Although the change in average...

Convolution: Math, Graphics, and Discrete Signals

Convolution: Math, Graphics, and Discrete Signals

In any LTI (Linear Time-Invariant) system, the convolution of two signals is denoted using a convolution operator, assuming all initial conditions are zero. The convolution integral can be divided into two parts: the zero-input or natural response and the zero-state or forced response, with t0 indicating the initial time.
To simplify the convolution integral, it is assumed that both the input signal and impulse response are zero for negative time values. The graphical convolution process...

Observational Learning

Observational Learning

Albert Bandura's observational learning, also known as imitation or modeling, occurs when a person observes and imitates another's behavior. It is a quicker process than operant conditioning. A well-known example is the Bobo doll study, where children who saw an adult acting aggressively towards the doll were more likely to act aggressively when left alone, compared to those who observed a nonaggressive adult. Many psychologists view observational learning as a form of latent learning...

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by

Same author

Learning in Feedforward Neural Networks Accelerated by Transfer Entropy.

Entropy (Basel, Switzerland)·2020

Same author

Semiotic Aggregation in Deep Learning.

Entropy (Basel, Switzerland)·2020

Same author

Transfer Information Energy: A Quantitative Indicator of Information Transfer between Time Series.

Entropy (Basel, Switzerland)·2020

See all related articles

Search research articles

Related Experiment Videos

Learning in Convolutional Neural Networks Accelerated by Transfer Entropy.

Adrian Moldovan^1,2, Angel Caţaron^1,2, Răzvan Andonie^1,3

¹Department of Electronics and Computers, Transilvania University, 500024 Braşov, Romania.

Entropy (Basel, Switzerland)

|September 28, 2021

Summary

This summary is machine-generated.

Transfer Entropy (TE) integration accelerates Convolutional Neural Network (CNN) training by quantifying neuron communication. This method optimizes learning by focusing on key neuron pairs, offering a stable, periodic feedback mechanism.

Keywords:

Convolutional Neural Network causality deep learning transfer entropy

Related Experiment Videos

Area of Science:

Artificial Intelligence
Machine Learning
Computational Neuroscience

Background:

Growing interest in Transfer Entropy (TE) for quantifying effective connectivity in artificial neural networks.
TE's potential in analyzing relationships between neuron outputs across different layers in feedforward networks.

Purpose of the Study:

To integrate Transfer Entropy (TE) into the learning mechanisms of Convolutional Neural Network (CNN) architectures.
To introduce a novel training mechanism incorporating TE feedback connections for enhanced CNN performance.

Main Methods:

Developed a novel training mechanism for CNNs that integrates TE feedback connections.
Experimented with CNN classifiers, focusing TE analysis on neuron pairs between the last two fully connected layers.
Treated TE as a periodic, slowly changing meta-parameter, acting as a smoothing factor.

Main Results:

Integrating TE feedback accelerates CNN training, requiring fewer epochs.
A reasonable computational overhead-accuracy trade-off was achieved by focusing TE on specific neuron pairs.
TE provides stability and acts as a periodic smoothing factor during training.

Conclusions:

The proposed TE-integrated training mechanism enhances CNN training speed and stability.
Selective application of TE, particularly between the last two fully connected layers, optimizes performance and manages computational cost.
TE functions as a valuable, slowly evolving meta-parameter for improving CNN learning dynamics.