Optoelectronic nonlinear Softmax operator based on diffractive neural networks.

Area of Science:

• Optoelectronics
• Deep Learning
• Computational Science

Background:

• The Softmax function is fundamental to many statistical and deep learning (DL) models, including large language models like ChatGPT.
• Computing Softmax is computationally intensive, particularly for large-scale applications, hindering model efficiency and scalability.
• Existing software and hardware acceleration strategies for Softmax often lack sufficient efficiency and scalability.

Purpose of the Study:

• To propose and demonstrate a photonic-computing approach for efficient and scalable Softmax computation.
• To address the computational bottlenecks associated with the Softmax operation in deep learning.

Main Methods:

• Development of a photonic-computing system featuring massive programmable neurons.
• Utilizing diffraction-based computation for executing the Softmax operation.
• Experimental validation of the system's performance and generalization capabilities.

Main Results:

• The photonic-computing system accurately computes the Softmax operation with high efficiency and scalability.
• Experimental results demonstrate a mean square error below 10^-5 across diverse tasks.
• The system exhibits robust performance even under realistic operational constraints.

Conclusions:

• The proposed photonic-computing solution offers an accurate, efficient, and scalable method for Softmax computation.
• This approach can optimize the Softmax mechanism and inspire new optoelectronic accelerators.
• The system shows promise as a plug-and-play module for general optoelectronic acceleration applications.