Related Experiment Videos
Linear independence of internal representations in multilayer perceptrons
1Harvard/MIT Division of Health Sciences and Technology, Massachusetts Institute of Technology, Cambridge, MA 02139, USA.
IEEE Transactions on Neural Networks
|February 7, 2008
Summary
Linear independence in multilayer perceptron internal representations is key for exact learning. Sigmoidal activations enable this, minimizing hidden units needed for accurate pattern recognition.
Area of Science:
- Artificial Intelligence
- Machine Learning
- Neural Networks
Background:
- Multilayer perceptrons (MLPs) are fundamental neural network architectures.
- Exact learning in MLPs remains a challenge, particularly concerning internal representations.
Purpose of the Study:
- To identify essential properties of MLP internal representations for achieving exact learning.
- To analyze the role of activation functions and training algorithms in this process.
Main Methods:
- Investigated the linear independence of internal representations in MLPs.
- Analyzed the output properties of sigmoidal hidden unit activation functions.
- Examined the relationship between input patterns, their rank, and the minimum number of hidden units.
Main Results:
- Linear independence of internal representations is identified as crucial for exact learning.
- Sigmoidal activation functions generate linearly independent outputs.
- The minimum hidden units required equals the number of patterns minus the input pattern rank.
Conclusions:
- The study establishes linear independence as a necessary condition for exact MLP learning.
- Sigmoidal activations facilitate achieving this condition.
- Training algorithms can enhance linear independence, improving learning accuracy.
Related Concept Videos
01:27Continuity for Functions of Multiple Variables
Continuity in multivariable functions extends the concept familiar from single-variable calculus into higher dimensions, where a function's output depends on two or more input variables. This generalization is crucial in modeling real-world phenomena across spatial domains. A multivariable function is considered continuous at a point if three conditions are simultaneously satisfied: the function is defined at that point, the limit of the function exists as the input approaches the point from...
01:22Multi-input and Multi-variable systems
Cruise control systems in cars are designed as multi-input systems to maintain a driver's desired speed while compensating for external disturbances such as changes in terrain. The block diagram for a cruise control system typically includes two main inputs: the desired speed set by the driver and any external disturbances, such as the incline of the road. By adjusting the engine throttle, the system maintains the vehicle's speed as close to the desired value as possible.
In the absence of...
In the absence of...
01:23Linear time-invariant Systems
A system is linear if it displays the characteristics of homogeneity and additivity, together termed the superposition property. This principle is fundamental in all linear systems. Linear time-invariant (LTI) systems include systems with linear elements and constant parameters.
The input-output behavior of an LTI system can be fully defined by its response to an impulsive excitation at its input. Once this impulse response is known, the system's reaction to any other input can be calculated...
The input-output behavior of an LTI system can be fully defined by its response to an impulsive excitation at its input. Once this impulse response is known, the system's reaction to any other input can be calculated...
01:26Classification of Systems-I
Linearity is a system property characterized by a direct input-output relationship, combining homogeneity and additivity.
Homogeneity dictates that if an input x(t) is multiplied by a constant c, the output y(t) is multiplied by the same constant. Mathematically, this is expressed as:
Homogeneity dictates that if an input x(t) is multiplied by a constant c, the output y(t) is multiplied by the same constant. Mathematically, this is expressed as:
01:27Associative Learning
Associative learning is a fundamental concept in behavioral psychology, wherein a connection is established between two stimuli or events, leading to a learned response. This process is critical in understanding how behaviors are acquired and modified. Conditioning, the mechanism through which associations are formed, can be divided into two main types: classical conditioning and operant conditioning, each elucidating different aspects of associative learning.
Classical conditioning, also known...
Classical conditioning, also known...
01:20Convolution Properties I
Convolution computations can be simplified by utilizing their inherent properties.
The commutative property reveals that the input and the impulse response of an LTI (Linear Time-Invariant) system can be interchanged without affecting the output:
The commutative property reveals that the input and the impulse response of an LTI (Linear Time-Invariant) system can be interchanged without affecting the output: