Probabilistic Deterministic Finite Automata and Recurrent Networks, Revisited

Area of Science:

• Computational neuroscience
• Machine learning theory
• Information theory

Background:

• Recurrent neural networks (RNNs) and reservoir computers (RCs) theoretically mimic finite-state automata.
• Understanding their practical predictive capabilities, especially in data-limited scenarios, remains an active research area.

Purpose of the Study:

• To evaluate the predictive performance of generalized linear models, RCs, and Long Short-Term Memory (LSTM) RNNs.
• To assess these models' ability to predict stochastic processes from probabilistic deterministic finite-state automata (PDFA) in the small-data limit.
• To analyze predictive accuracy and information-theoretic feature extraction using a rate-distortion curve.

Main Methods:

• Systematic enumeration of PDFAs to create benchmark datasets.
• Generation of stochastic processes from PDFAs with known randomness and correlation structures.
• Computation of optimal memory-limited predictors for PDFAs.
• Testing generalized linear models, RCs, and LSTMs on PDFA-generated data.
• Evaluation using predictive accuracy and distance to the predictive rate-distortion curve.

Main Results:

• Long Short-Term Memory (LSTM) recurrent neural networks (RNNs) showed a surprising decrease in predictive accuracy with limited data.
• LSTMs demonstrated superior performance in lossy predictive feature extraction, as measured by their proximity to the rate-distortion curve.
• Generalized linear models and reservoir computers were also evaluated against these metrics.

Conclusions:

• The study highlights the trade-offs between predictive accuracy and information-theoretic feature extraction in RNNs under data scarcity.
• Causal states are identified as a valuable concept for understanding the predictive capabilities of RNNs.
• The findings underscore the importance of considering model architecture and data availability when assessing predictive performance.