Fourier multi-component and multi-layer neural networks: Unlocking high-frequency potential.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Neural Network Architectures

Background:

• Neural network performance hinges on architecture and activation function choice.
• Matching these elements is crucial for effective representation and learning.
• Standard networks face challenges with high-frequency function approximation.

Purpose of the Study:

• Introduce the Fourier Multi-Component and Multi-Layer Neural Network (FMMNN).
• Analyze FMMNNs' expressive power, optimization landscape, and training dynamics.
• Demonstrate FMMNNs' effectiveness on oscillatory function approximation tasks.

Main Methods:

• FMMNN combines sine-type activations with multi-component, multi-layer structures.
• Components are trainable linear combinations of fixed random sine basis functions.
• Multi-layer composition enables adaptive high-frequency feature generation.

Main Results:

• FMMNNs exhibit exponential expressive power for function approximation, even with low-rank structures.
• Optimization landscape is more favorable than standard fully connected networks, especially for high-frequency targets.
• A scaled random initialization method accelerates training and enhances performance.

Conclusions:

• FMMNNs offer significant advantages in approximating oscillatory functions.
• The proposed architecture and initialization method lead to strong accuracy and favorable convergence.
• FMMNNs represent a promising advancement in neural network design for specific tasks.