Related Experiment Video
Updated: Jul 19, 2026
11:54
Real-Time Proxy-Control of Re-Parameterized Peripheral Signals using a Close-Loop Interface
Published on: May 8, 2021
Orthogonality is not a panacea: backpropagation and "catastrophic interference"
1Department of Educational Psychology, Waseda University, Tokyo, Japan. yamag-psy@toki.waseda.jp
Scandinavian Journal of Psychology
|September 22, 2006
Summary
Not all orthogonal inputs prevent catastrophic forgetting equally in connectionist models. Input coding schemes and network size interact, influencing the severity of interference during sequential training.
Area of Science:
- Artificial Intelligence
- Machine Learning
- Computational Neuroscience
Background:
- Connectionist models using backpropagation learning are prone to catastrophic interference (forgetting) during sequential training.
- Orthogonal inputs have been proposed to mitigate this interference, but their effectiveness across different schemes is not fully understood.
Purpose of the Study:
- To rigorously assess whether all orthogonal input coding schemes equally reduce catastrophic interference.
- To investigate the influence of network size on interference levels with different orthogonal input schemes.
Main Methods:
- Employed a rigorous assessment methodology to evaluate interference.
- Compared three distinct orthogonal input coding schemes.
- Analyzed results across varying network sizes.
Main Results:
- Significant differences in interference levels were observed between the coding schemes.
- Dense orthogonal inputs led to more severe interference than sparse inputs in larger networks.
- In smaller networks, all three coding schemes resulted in comparable interference levels.
Conclusions:
- Orthogonal inputs do not universally eliminate catastrophic interference; effectiveness varies by coding scheme.
- The interaction between input coding strategy and network size critically determines the extent of interference.
Related Concept Videos
01:10Interference: Path Lengths
Consider two sources of sound, that may or may not be in phase, emitting waves at a single frequency, and consider the frequencies to be the same.
Two special sources may be considered when they are in phase. This can be easily achieved by feeding the two sources from the same source. An example would be synchronizing the two speakers by feeding them with the same source, such as the sound waves produced by a tuning fork. This setup ensures that the two sources have the same frequency and are...
Two special sources may be considered when they are in phase. This can be easily achieved by feeding the two sources from the same source. An example would be synchronizing the two speakers by feeding them with the same source, such as the sound waves produced by a tuning fork. This setup ensures that the two sources have the same frequency and are...
01:07Interference and Superposition of Waves
When two waves of the same nature occur in the same region simultaneously, they result in interference. Interference of waves implies that the net effect of the waves is the sum of the individual waves' effects. However, it does not imply that the individual waves affect the propagation of other waves.
Interference occurs in mechanical waves, such as sound waves, waves on a string, and surface water waves. Mechanical waves correspond to the physical displacement of particles. Hence,...
Interference occurs in mechanical waves, such as sound waves, waves on a string, and surface water waves. Mechanical waves correspond to the physical displacement of particles. Hence,...
00:53Sound Waves: Interference
Sound waves can be modeled either as longitudinal waves, wherein the molecules of the medium oscillate around an equilibrium position, or as pressure waves. When two identical waves from the same source superimpose on each other, the combination of two crests or two troughs results in amplitude reinforcement known as constructive interference. If two identical waves, that are initially in phase, become out of phase because of different path lengths, the combination of crests with troughs...
00:59Propagation of Uncertainty from Random Error
An experiment often consists of more than a single step. In this case, measurements at each step give rise to uncertainty. Because the measurements occur in successive steps, the uncertainty in one step necessarily contributes to that in the subsequent step. As we perform statistical analysis on these types of experiments, we must learn to account for the propagation of uncertainty from one step to the next. The propagation of uncertainty depends on the type of arithmetic operation performed on...
01:10Propagation of Uncertainty from Systematic Error
The atomic mass of an element varies due to the relative ratio of its isotopes. A sample's relative proportion of oxygen isotopes influences its average atomic mass. For instance, if we were to measure the atomic mass of oxygen from a sample, the mass would be a weighted average of the isotopic masses of oxygen in that sample. Since a single sample is not likely to perfectly reflect the true atomic mass of oxygen for all the molecules of oxygen on Earth, the mass we obtain from this particular...
01:24Effects of feedback
Feedback in control systems plays a critical role in shaping various operational parameters, extending beyond simple error reduction to influence stability, bandwidth, gain, impedance, and sensitivity. Understanding these effects requires examining a basic feedback system characterized by defined input, output, error, and feedback signals.
Feedback significantly modifies the gain of a control system. The gain of a system without feedback is altered by a factor of one plus GH, where G represents...
Feedback significantly modifies the gain of a control system. The gain of a system without feedback is altered by a factor of one plus GH, where G represents...