Complexity-calibrated benchmarks for machine learning reveal when prediction algorithms succeed and mislead.

Area of Science:

• Computational Neuroscience
• Machine Learning
• Time Series Analysis

Background:

• Recurrent neural networks (RNNs) are widely used for time series forecasting across various domains like finance, climate, and language.
• Reservoir computers are a simplified, easily trainable class of RNNs.
• A recent 'next-generation' reservoir computer model features a finite-past memory trace.

Purpose of the Study:

• To investigate the inherent limitations of finite-past memory traces in next-generation reservoir computers.
• To assess the performance of current RNNs in predicting complex, non-Markovian processes.
• To identify requirements for future RNN architectures.

Main Methods:

• Utilized Fano's inequality to establish a lower bound on prediction error for next-generation reservoir computers.
• Analyzed highly non-Markovian processes generated by large probabilistic state machines.
• Presented concentration-of-measure results for complex, randomly generated processes.

Main Results:

• Next-generation reservoir computers exhibit a prediction error significantly higher than the theoretical minimum for complex processes.
• Popular RNNs demonstrate suboptimal performance in predicting intricate time series.
• Large probabilistic state machines, particularly -machines, are crucial for generating unbiased training data.

Conclusions:

• Finite-past memory traces impose fundamental limitations on reservoir computer prediction accuracy.
• Current RNNs are not optimized for highly complex, non-Markovian time series.
• Development of novel, optimized RNN architectures is essential.
• Large probabilistic state machines serve as vital benchmarks for evaluating RNNs.