A refined lion optimizer for deep learning.

Area of Science:

• Deep Learning
• Optimization Algorithms
• Computer Vision

Background:

• Optimization algorithms are crucial for training neural networks, influencing weight updates, learning rates, and loss.
• The Lion optimizer offers faster training and memory efficiency but can face non-convergence issues due to its discrete sign function.
• Existing optimizers struggle with dynamic adaptation of momentum and parameter updates in complex models.

Purpose of the Study:

• Introduce the Refined Lion Optimizer (RLion) to address the limitations of the Lion optimizer.
• Enhance parameter update adaptivity and model convergence reliability.
• Evaluate RLion's performance across various deep learning tasks including classification, object detection, and semantic segmentation.

Main Methods:

• Developed RLion with a novel update rule using a non-linear continuous bounded function for momentum and scaling factor.
• Theoretically analyzed RLion's convergence properties and stability.
• Empirically validated RLion by training models like FasterNet, EfficientNetV2, YOLOV8, YOLOV11, DeepLabV3+, TwinLiteNet, and UNet on diverse datasets (ImageNet1k, VOC2012, Caltech 101, etc.).

Main Results:

• RLion demonstrated improved classification accuracy, outperforming AdamW by approximately [Formula: see text] on several models.
• For object detection and semantic segmentation, RLion achieved performance comparable to AdamW.
• RLion effectively mitigated the gradient explosion/disappearance issues inherent in the Lion optimizer.

Conclusions:

• RLion offers superior convergence performance and reliability compared to Lion and AdamW.
• The novel update rule enables adaptive parameter adjustments, leading to more stable and efficient training.
• RLion presents a versatile and effective optimization solution for a wide range of deep learning applications.