OSLNet: Deep Small-Sample Classification with an Orthogonal Softmax Layer.

Area of Science:

• Computer Science
• Machine Learning
• Artificial Intelligence

Background:

• Deep neural networks (DNNs) with multiple nonlinear layers create large function spaces, increasing the risk of overfitting, especially with small-sample datasets.
• Mitigating overfitting in small-sample classification necessitates learning more discriminative features.

Purpose of the Study:

• To identify a neural network subspace that promotes a large decision margin, thereby reducing overfitting.
• To introduce and validate the Orthogonal Softmax Layer (OSL) for improved small-sample classification.

Main Methods:

• Proposing the Orthogonal Softmax Layer (OSL) that enforces orthogonality of weight vectors in the classification layer during training and testing.
• Analyzing the Rademacher complexity of networks utilizing OSL, demonstrating a reduction to 1/K compared to fully connected layers, where K is the number of classes.
• Conducting experiments on four small-sample benchmark datasets to evaluate OSL performance.

Main Results:

• The proposed OSL demonstrates superior performance compared to existing methods on small-sample benchmark datasets.
• The Rademacher complexity analysis indicates a tighter generalization error bound for networks using OSL.
• OSL shows applicability and effectiveness even on large-sample datasets.

Conclusions:

• The Orthogonal Softmax Layer (OSL) is an effective method for mitigating overfitting in small-sample classification tasks.
• OSL contributes to improved generalization by facilitating a larger decision margin and reducing model complexity.
• The proposed method offers a valuable advancement for deep learning applications dealing with limited data and shows potential for broader use.