Representational Gradient Boosting: Backpropagation in the Space of Functions

Area of Science:

• Machine Learning
• Artificial Intelligence
• Function Estimation

Background:

• Artificial neural networks (ANNs) are popular for representational learning, estimating functions of functions.
• Current meta-learning approaches combine models independently after training.

Purpose of the Study:

• Introduce Representational Gradient Boosting (RGB), a nonparametric algorithm for estimating nested functions.
• Demonstrate RGB's ability to jointly optimize and combine diverse function classes like Neural Networks (NN) and Gradient Boosting (GB).

Main Methods:

• RGB utilizes backpropagation in the function space to estimate multi-layer architectures.
• It functions as an optimized stacking procedure, learning to combine models.
• RGB does not assume predefined functional forms in its nodes or output.

Main Results:

• RGB offers advantages over current meta-learning by providing optimized, joint stacking.
• RGB outperforms traditional Gradient Boosting (GB) on problems with high-order interactions due to its nested structure.
• The study provides theoretical and practical insights into recovering nested functions.

Conclusions:

• RGB represents a novel approach to representational learning by estimating nested functions.
• Joint optimization and model combination in RGB address limitations of independent model stacking.
• RGB shows significant improvements in handling complex, high-order interactions in data.