Related Experiment Video
Updated: May 13, 2026
05:55
Modeling the Functional Network for Spatial Navigation in the Human Brain
Published on: October 13, 2023
Local minima in hierarchical structures of complex-valued neural networks
1Mathematical Neuroinformatics Group, Human Technology Research Institute, National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba Central 2, 1-1-1 Umezono, Tsukuba, Ibaraki, 305-8568, Japan.
Summary
Complex-valued neural networks resolve local minima issues common in real-valued networks. By transforming to complex numbers, most critical points become saddle points, improving neural network learning.
Area of Science:
- Artificial Intelligence
- Machine Learning
- Deep Learning
Background:
- Hierarchical structures in real-valued neural networks often lead to local minima.
- These local minima create plateaus, negatively impacting the learning process.
- Previous research showed that critical points in real-valued networks with H-1 neurons generate numerous critical points in networks with H neurons.
Purpose of the Study:
- To investigate the impact of extending real-valued neural networks to complex numbers.
- To analyze the nature of critical points in complex-valued neural networks compared to real-valued ones.
- To determine if complex-valued neural networks can mitigate issues caused by local minima.
Main Methods:
- Theoretical analysis of critical points in real-valued and complex-valued neural networks.
- Mathematical extension of real-valued neural network models to complex domains.
- Comparison of the distribution and properties of critical points in both network types.
Main Results:
- Extending real-valued neural networks to complex numbers effectively resolves many local minima.
- Most critical points in complex-valued neural networks are saddle points, unlike in real-valued networks.
- This shift from local minima to saddle points is a key characteristic of complex-valued neural networks.
Conclusions:
- Complex-valued neural networks offer a promising approach to overcome learning challenges posed by local minima.
- The prevalence of saddle points in complex-valued networks suggests improved optimization landscapes.
- This work highlights a significant advantage of complex-valued neural networks for deep learning applications.
Related Concept Videos
01:18Hierarchy of Motor Control
The hierarchy of motor control refers to the different levels of organization and processing involved in controlling movement in the body. These levels range from higher cortical areas involved in planning and decision-making to lower spinal cord reflexes that respond automatically to external stimuli.
01:25Neural Circuits
Neural circuits and neuronal pools are two of the main structures found in the nervous system. Neural circuits are networks of neurons that work together to carry out a specific task or process. They consist of interconnected neurons and glial cells, which provide structural and metabolic support.
Neuronal pools are collections of nerve cells with similar functions and interact through chemical and electrical signals. These pools include both interneurons (the central neural circuit nodes that...
Neuronal pools are collections of nerve cells with similar functions and interact through chemical and electrical signals. These pools include both interneurons (the central neural circuit nodes that...
01:37Neural Regulation
Digestion begins with a cephalic phase that prepares the digestive system to receive food. When our brain processes visual or olfactory information about food, it triggers impulses in the cranial nerves innervating the salivary glands and stomach to prepare for food.
01:22Absolute and Local Extreme Values
The highest and lowest values of a function, relative to a reference axis, are known as extreme values. These include absolute maximum and absolute minimum values, which represent the highest and lowest points the function reaches across its entire domain. Within a restricted portion of the function, the highest and lowest values are referred to as local maximum and local minimum values, respectively.Periodic functions, such as sine and cosine, show extreme values at infinitely many points due...
01:26Linear Approximation in Frequency Domain
Linear systems are characterized by two main properties: superposition and homogeneity. Superposition allows the response to multiple inputs to be the sum of the responses to each individual input. Homogeneity ensures that scaling an input by a scalar results in the response being scaled by the same scalar.
In contrast, nonlinear systems do not inherently possess these properties. However, for small deviations around an operating point, a nonlinear system can often be approximated as linear.
In contrast, nonlinear systems do not inherently possess these properties. However, for small deviations around an operating point, a nonlinear system can often be approximated as linear.
01:10Energy Diagrams - II
Energy diagrams are important to understand the dynamics of a system. The topology of an energy diagram helps illustrate the equilibrium points of the system.
The point in the energy diagram at which the system’s potential energy is the lowest is known as the local minima. The system tends to stay in this position indefinitely unless acted upon by a net force. The slope of the potential energy diagram at the local minima is zero, indicating that zero net force is acting on the system. The slope...
The point in the energy diagram at which the system’s potential energy is the lowest is known as the local minima. The system tends to stay in this position indefinitely unless acted upon by a net force. The slope of the potential energy diagram at the local minima is zero, indicating that zero net force is acting on the system. The slope...