Statistical mechanics of support vector regression.

Area of Science:

• Computational Neuroscience
• Deep Learning
• Machine Learning Theory

Background:

• Relating geometric properties of neural representations to task performance is a key challenge in deep learning and computational neuroscience.
• Neural variability can impact the precision of continuous decoding tasks.

Purpose of the Study:

• To investigate the relationship between neural representation geometry and task performance in continuous decoding.
• To analyze the role of the tolerance parameter (ɛ) in support vector regression for understanding linear decodability and neural variability.

Main Methods:

• Utilized statistical mechanics to study average-case learning curves for ɛ-insensitive support vector regression.
• Analyzed the capacity of support vector regression as a measure of linear decodability.
• Validated theoretical predictions using toy models and deep neural networks.

Main Results:

• Identified a phase transition in training error at a critical load, demonstrating the interplay between the tolerance parameter (ɛ) and neural variability.
• Discovered a double-descent phenomenon in generalization error, where ɛ acts as a regularizer, influencing peak suppression and shifting.
• Extended support vector machine theory to continuous tasks with inherent neural variability.

Conclusions:

• The tolerance parameter (ɛ) plays a crucial role in regularizing generalization error and managing neural variability in continuous decoding tasks.
• Support vector regression provides a valuable framework for understanding linear decodability in the presence of neural variability.