Deep Residual Echo State Networks: Exploring Residual Orthogonal Connections in Untrained Recurrent Neural Networks

Echo state networks (ESNs) are a particular type of untrained recurrent neural networks (RNNs) within the reservoir computing (RC) framework, popular for their fast and efficient learning. However, traditional ESNs often struggle with long-term information processing. In this article, we introduce a novel class of deep untrained RNNs based on temporal residual connections, called deep residual ESNs (DeepResESNs). We show that leveraging a hierarchy of untrained residual recurrent layers significantly boosts memory capacity and long-term temporal modeling. For the temporal residual connections, we consider different orthogonal configurations, including randomly generated and fixed-structure, and study their effect on network dynamics. Athorough mathematical analysis outlines necessary and sufficient conditions to ensure stable dynamics within DeepResESN. Empirically, the proposed approach consistently outperforms traditional shallow and deep RC on a variety of time-series tasks. Overall, DeepResESN offers a promising approach for designing hierarchical ESNs with better prediction accuracy on long sequences, without sacrificing the computational advantages that make RC attractive.