Related Experiment Videos
The effects of the irregular sample and missing data in time series analysis
David M Kreindler1, Charles J Lumsden
1Sunnybrook & Women's College, Health Science Centre, 2075 Bayview Ave. Rm. FG-17, Toronto, Ontario, M4N 3M5, Canada. david.kreindler@utoronto.ca
Nonlinear Dynamics, Psychology, and Life Sciences
|March 8, 2006
Summary
Analyzing irregularly sampled human self-report time series data is challenging. This study shows that repairing up to 15% missing data using methods like local interpolation can enable analysis with standard techniques.
Area of Science:
- Data Science
- Complex Systems Analysis
- Time Series Analysis
Background:
- Human self-report time series data often exhibit irregular sampling rates due to natural data generation processes.
- Existing methods for analyzing such irregularly sampled data are limited.
- Irregular sampling can hinder the application of standard time series analysis tools.
Purpose of the Study:
- To assess the utility of different data patching techniques for handling irregularly sampled time series data.
- To compare the performance of nonlinear analytical tools on complete versus patched data sets.
- To determine the feasibility of repairing data for analysis with techniques assuming regular sampling.
Main Methods:
- Synthetic time series data representing quasiperiodic, chaotic, and self-organized critical dynamics were generated.
- Regularly sampled data were intentionally disrupted by data point removal or stochastic time shifts.
- Missing data segments were reconstructed using segment concatenation, average value filling, or local phase space interpolation.
- Nonlinear analytical tools (autocorrelations, correlation dimensions) and spectral analysis (power spectra, Lomb periodograms) were applied to compare complete and patched data.
Main Results:
- Local interpolation in phase space effectively preserved key data features but required substantial intact data.
- Segment concatenation and average value filling did not require large intact data segments but introduced distortions.
- Analysis of irregularly sampled data sets with up to 15% missing data, after repair, showed minimal substantial errors compared to intact series.
- Lomb periodograms were effective for analyzing decimated (irregularly sampled) data.
Conclusions:
- Repairing irregularly sampled time series data, particularly with up to 15% missing data, is feasible for analysis using standard techniques.
- The choice of data patching method impacts the accuracy of subsequent analyses; local interpolation is effective but data-intensive.
- Further research can build upon these findings to develop robust methods for analyzing real-world, irregularly sampled datasets.