Using cross-validation methods to select time series models: Promises and pitfalls.

Area of Science:

• Psychology
• Statistics
• Time Series Analysis

Background:

• Vector autoregressive (VAR) modeling is common in psychology for time series analysis.
• Short time series in psychological studies often lead to VAR model overfitting and poor prediction.
• Cross-validation (CV) is recommended for assessing model predictive ability, but its performance with psychological time series data is not well understood.

Purpose of the Study:

• To examine how 10-fold CV and blocked CV estimate prediction errors for person-mean, AR, and VAR models.
• To evaluate the impact of data characteristics on CV method performance.
• To compare CV methods against Akaike (AIC) and Bayesian (BIC) information criteria for model selection.

Main Methods:

• Simulation study analyzing prediction errors of three time series models (person-mean, AR, VAR).
• Evaluation of two cross-validation techniques: 10-fold CV and blocked CV.
• Comparison of CV methods with traditional model selection criteria (AIC, BIC).

Main Results:

• CV methods showed a tendency to underestimate prediction errors for simpler models (person-mean, AR).
• CV methods tended to overestimate prediction errors for VAR models, particularly with small sample sizes.
• Blocked CV generally outperformed AIC and BIC in selecting the most predictive models.

Conclusions:

• Cross-validation, especially blocked CV, is a valuable tool for assessing time series model predictive accuracy in psychology.
• CV methods offer advantages over AIC and BIC for model selection, despite potential biases with small sample sizes.
• Guidelines are provided for practical application of CV in psychological time series analyses.