SmooNet: Smooth operator neural network and functional differential equation

Dynamical systems are often modeled by differential equations, where the ordinary differential equations (ODEs) are most commonly used. One major limitation of the ODE model is that it assumes the derivatives of the system only depend on the concurrent values of the state. This assumption can oversimplify the mechanisms of dynamical systems and fail to model the memory effects which are also common in dynamical systems. To address this limitation, we propose a general Functional Differential Equation (FDE) model which allows the derivative to explicitly depend on both the current value and a historical segment of the system through an unknown operator defined in the space of all continuous functions on an interval. We propose a family of operators, called the Smooth Operator Neural Network (SmooNet), to approximate the operator in FDE. Distinct from the commonly used neural networks with scalar inputs, a unique feature of the SmooNet is that its hidden layer is a continuum with infinite number of nodes, which we call a hidden string, and the complexity of SmooNet is completely determined by the smoothness of the learnable functions. We establish not only that the SmooNet can universally approximate the operator in FDE, but also that the solution to the approximate neural FDE can be uniformly and arbitrarily close to the solution of the original FDE. We propose a novel optimization problem based on moving windows to construct the SmooNet, and then make forecasts using the neural FDE. Through empirical studies, we show that the new model and method provide a flexible and efficient tool to study dynamical systems and make forecasts.