The Data Efficiency of Deep Learning Is Degraded by Unnecessary Input Dimensions

Area of Science:

• Artificial Intelligence
• Computational Neuroscience
• Machine Learning

Background:

• Biological learning systems exhibit superior data efficiency compared to Deep Neural Networks (DNNs).
• The reasons for this data efficiency gap between biological and artificial intelligence remain unclear.
• Biological systems utilize mechanisms like foveation to filter irrelevant input dimensions, unlike typical DNNs.

Purpose of the Study:

• To investigate the impact of unnecessary input dimensions on the data efficiency of DNNs.
• To determine if task-unrelated input dimensions hinder DNNs' ability to learn from limited data.
• To explore the potential benefits of input dimension reduction mechanisms for DNNs.

Main Methods:

• The study analyzed the effect of task-unrelated input dimensions on DNN generalization performance.
• Experiments were conducted to quantify the amount of training data required under varying input dimension conditions.
• The research focused on the input layer's sensitivity to extraneous data, distinct from hidden layers.

Main Results:

• Unnecessary input dimensions significantly degrade the data efficiency of DNNs.
• Task-unrelated input dimensions require more training examples for DNNs to achieve desired performance.
• DNNs' performance is negatively impacted by the presence of extraneous information in the input data.

Conclusions:

• Mechanisms that remove task-unrelated input dimensions are crucial for enhancing DNN data efficiency.
• Implementing strategies similar to biological foveation could lead to significant gains in machine learning.
• Reducing input dimensionality is a key factor in bridging the data efficiency gap between AI and biological learning.