Contrastive Forward-Forward: A training algorithm of vision transformer.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Neuroscience

Background:

• Backpropagation is the standard for training artificial neural networks.
• Forward-Forward (FF) is a novel, biologically plausible alternative but has performance limitations.
• Current FF algorithms are evaluated on simple networks for image classification.

Purpose of the Study:

• Extend the FF algorithm to complex Vision Transformer (ViT) networks.
• Improve FF algorithm's performance and biological plausibility.
• Introduce a novel variant, Contrastive Forward-Forward (CFF).

Main Methods:

• Modified the FF algorithm by incorporating contrastive learning principles.
• Applied the modified algorithm to Vision Transformer architectures.
• Evaluated CFF against baseline FF and backpropagation.

Main Results:

• CFF significantly outperforms baseline FF, improving accuracy by up to 10%.
• CFF accelerates convergence speed by 5 to 20 times compared to baseline FF.
• CFF reduces the performance gap with backpropagation, especially under inaccurate supervision.

Conclusions:

• Contrastive Forward-Forward is a promising, biologically plausible training method for advanced neural networks.
• CFF demonstrates superior performance and efficiency over baseline FF.
• The CFF approach offers a competitive alternative to backpropagation in specific machine learning tasks.