ResNEsts and DenseNEsts: Block-based DNN Models with Improved Representation Guarantees.

Area of Science:

• Computer Science
• Machine Learning
• Deep Learning

Background:

• Standard Residual Networks (ResNets) are widely used in computer vision.
• Previous literature models proving ResNets superior to linear predictors differ from standard ResNets.
• These models lack final nonlinearities, impacting their properties.

Purpose of the Study:

• Define and analyze Residual Nonlinear Estimators (ResNEsts) by modifying standard ResNets.
• Investigate the training properties and performance guarantees of ResNEsts.
• Propose and analyze Densely connected Nonlinear Estimators (DenseNEsts) inspired by DenseNets.

Main Methods:

• Defined ResNEsts by removing nonlinearities from the final residual representation in standard ResNets.
• Introduced augmented ResNEsts (A-ResNEsts) to decouple prediction weights from basis learning.
• Demonstrated that DenseNEsts can be represented as wide ResNEsts with bottleneck blocks.

Main Results:

• Wide ResNEsts with bottleneck blocks guarantee performance improvement with added layers.
• A-ResNEsts establish empirical risk lower bounds for ResNEsts.
• ResNEsts exhibit diminishing feature reuse, which can be mitigated by widening the input space.
• DenseNEsts inherently possess desirable training properties without architectural modifications.

Conclusions:

• ResNEsts offer a distinct perspective on residual network properties, particularly regarding linear estimation.
• Architectural modifications like augmentation or widening can address limitations in ResNEsts.
• DenseNEsts provide a robust alternative, achieving desirable properties inherently.