Initializing photonic feed-forward neural networks using auxiliary tasks.

Area of Science:

• Photonics
• Artificial Intelligence
• Deep Learning

Background:

• Photonic accelerators offer fast and energy-efficient Deep Learning (DL) implementations.
• Current photonic DL accelerators often lack support for common activation functions like ReLU, using alternatives that cause training instability.
• Training photonic DL models is challenging due to vanishing gradients and the need for extensive hyper-parameter tuning.

Purpose of the Study:

• To propose a novel adaptive initialization method for photonic DL accelerators.
• To enhance the stability and ease of training for photonic DL models.
• To develop a method that is compatible with various photonic activation functions without requiring additional fine-tuning.

Main Methods:

• Developed a novel adaptive initialization technique utilizing auxiliary tasks.
• The method estimates optimal initialization variance for each network layer.
• Evaluated the approach using two datasets, two photonic activation functions, and three initialization methods.

Main Results:

• The proposed adaptive initialization significantly increases the stability of the training process.
• The method demonstrates effectiveness across different datasets and photonic activation functions.
• No further hyper-parameter fine-tuning is required when using the proposed method.

Conclusions:

• The novel adaptive initialization method effectively addresses training challenges in photonic DL.
• This approach enhances the practicality and applicability of photonic DL accelerators.
• The method offers a robust solution for stable and efficient training of photonic neural networks.