Comparative Study of Machine Learning and System Identification for Process Systems Engineering Dynamics

This study provides a comprehensive benchmarking of traditional system identification and modern machine learning (ML) models for the data-driven modeling of dynamical systems, with a focus on process systems engineering (PSE) applications. To achieve this, we deploy AutoSID, an automated end-to-end framework inspired by Machine Learning Operations (MLOps) principles. While AutoSID facilitates model selection, training, validation, and evaluation, its purpose here is to serve as a platform to investigate how well MLOps-inspired tools can be adapted for system identification tasks in PSE. Our investigation includes a comparison of 12 diverse model architectures from the system identification, machine learning, and deep learning literature, evaluated across 11 PSE case studies under varying data regimes. We employ four model search or hyperparameter optimization algorithms and three model selection criteria to ensure a thorough assessment. Our findings highlight the importance of model selection as the crucial step in system identification. Specifically, our results demonstrate the effectiveness of Bayesian optimization with tree-structured parzen estimators (TPE) for balanced model selection, while k-fold cross-validation proves to be a robust metric for performance evaluation during the selection process. In large-scale data scenarios, where performance differences between k-fold cross-validation and information criteria are small, information criteria emerge as a computationally efficient alternative. Once the "best" model structure is decided, in terms of model performance, we find that ML models with balanced complexity, such as tree ensemble models, consistently achieve superior predictive accuracy and computational efficiency, outperforming both simplistic and overly complex models. These findings provide actionable insights into model selection and performance evaluation for PSE practitioners and demonstrate the potential of incorporating MLOps-inspired workflows into the system identification process.