Related Experiment Videos
Analysis of ensemble learning using simple perceptrons based on online learning theory.
Seiji Miyoshi1, Kazuyuki Hara, Masato Okada
1Department of Electronic Engineering, Kobe City College of Technology, Gakuenhigashi-machi 8-3, Nishi-ku, Kobe 651-2194, Japan. miyoshi@kobe-kosen.ac.jp
Summary
Ensemble generalization error in online learning can be calculated using order parameters. AdaTron learning demonstrates superior "maintaining variety among students" for ensemble perceptron models.
Area of Science:
- Machine Learning
- Statistical Mechanics
- Computational Neuroscience
Background:
- Ensemble learning combines multiple models to improve generalization.
- Statistical learning theory aims to predict generalization error theoretically.
- Perceptron models are fundamental in neural network research.
Purpose of the Study:
- To theoretically calculate the ensemble generalization error of nonlinear perceptrons.
- To analyze the dynamics of order parameters in ensemble learning.
- To compare different learning rules for their suitability in ensemble methods.
Main Methods:
- Utilizing online learning and statistical mechanics frameworks.
- Defining and deriving differential equations for two key order parameters: teacher-student similarity and student-student similarity.
- Analytically solving these equations for Hebbian, perceptron, and AdaTron learning rules.
Main Results:
- Derived general differential equations for order parameter dynamics.
- Calculated ensemble generalization errors for three specific learning rules.
- Identified distinct affinities for ensemble learning, specifically in maintaining diversity among student models.
Conclusions:
- AdaTron learning exhibits a superior ability to maintain variety among student models compared to Hebbian and perceptron learning.
- The developed framework allows for the calculation and comparison of ensemble generalization errors.
- Findings contribute to understanding the theoretical underpinnings of ensemble methods in machine learning.