Correlations of Cross-Entropy Loss in Machine Learning

Area of Science:

• Machine Learning
• Deep Neural Networks
• Information Theory

Background:

• Cross-entropy loss is fundamental for training deep neural networks.
• Understanding relationships between loss functions can optimize training and feature extraction.

Purpose of the Study:

• To investigate novel correlations between cross-entropy and other divergence functions.
• To explore the relationship between cross-entropy and Euclidean distance over logits.
• To identify potential computational efficiencies and new feature representations.

Main Methods:

• Empirical observation of correlations between divergence functions.
• Mathematical analysis of related divergence functions and softmax outputs.
• Analysis of cross-entropy, triangular divergence, and Euclidean distance over logits.

Main Results:

• Demonstrated near-perfect correlation between cross-entropy and triangular divergence in certain scenarios.
• Showcased strong correlation between cross-entropy and Euclidean distance on softmax-derived logits.
• Established that logits can be effectively treated as features in a Euclidean space.

Conclusions:

• Triangular divergence presents a computationally cheaper alternative to cross-entropy loss.
• Euclidean distance over logits is a valid and synergistic measure of similarity for networks trained with softmax and cross-entropy.
• These findings offer practical implications for optimizing deep learning model training and analysis.