Related Experiment Video
Updated: Jul 30, 2025
10:46
A Method of Trigonometric Modelling of Seasonal Variation Demonstrated with Multiple Sclerosis Relapse Data
Published on: December 9, 2015
10.7K
Time series outlier removal and imputing methods based on Colombian weather stations data
Jaime Parra-Plazas1, Paulo Gaona-Garcia2, Leonardo Plazas-Nossa2
1Universidad Distrital Francisco José de Caldas, Carrera 7 No. 40B-53 piso 9, Bogotá, Colombia. jarrap@udistrital.edu.co.
Summary
Discrete Fourier Transform (DFT) effectively fills missing weather data, improving flood prediction models. This method handles data gaps and outliers with an average error of only 1%.
Area of Science:
- Hydrology and Environmental Science
- Data Science and Signal Processing
Background:
- Weather station time series are crucial for flood prediction models, providing data on flow and water levels.
- Data acquisition issues like outliers and missing values in weather time series hinder accurate simulation.
- A robust numerical strategy is needed to address data quality problems in weather time series analysis.
Purpose of the Study:
- To compare three time series analysis methods for evaluating multivariable processes offline.
- To propose and evaluate a numerical strategy for completing missing data in weather time series.
- To assess the effectiveness of Discrete Fourier Transform (DFT) in handling data gaps and outliers.
Main Methods:
- Applied Discrete Fourier Transform (DFT) for time series completion.
- Compared DFT with average and linear regression methods for filling missing data.
- Incorporated statistical analysis and outlier detection into the methodology.
Main Results:
- Discrete Fourier Transform (DFT) demonstrated effectiveness in managing various gap sizes and replacing missing values.
- The proposed DFT-based methodology resulted in low error percentages across all tested time series.
- An average error of 1% was achieved using DFT, indicating high accuracy in reconstructing time series patterns.
Conclusions:
- Discrete Fourier Transform (DFT) is a reliable numerical strategy for completing missing data in weather time series.
- DFT significantly improves the quality of data for flood analysis and simulation models.
- The method accurately reconstructs historical time series patterns, reducing the impact of data anomalies.
Related Concept Videos
01:17Precipitation and Co-precipitation
1.9K
Precipitation and coprecipitation methods can be used to separate a mixture of ions in a solution. In qualitative inorganic analysis, ions that form sparingly soluble precipitates with the same reagent are separated based on the differences in solubility products. For example, consider the separation of Cu(II) and Fe(II) ions by precipitation as insoluble sulfides. First, copper(II) sulfide is precipitated by the addition of acidic H2S, where the dissociation of H2S is suppressed. Adding H2S...
1.9K
01:08Outliers and Influential Points
4.1K
An outlier is an observation of data that does not fit the rest of the data. It is sometimes called an extreme value. When you graph an outlier, it will appear not to fit the pattern of the graph. Some outliers are due to mistakes (for example, writing down 50 instead of 500), while others may indicate that something unusual is happening. Outliers are present far from the least squares line in the vertical direction. They have large "errors," where the "error" or residual is the...
4.1K
01:19Precipitation Titration: Endpoint Detection Methods
1.9K
In argentometric precipitation titrations, endpoints can be detected visually by the Mohr, Volhard, and Fajans methods. In the Mohr method, adding a soluble chromate indicator gives an initial yellow color to the analyte solution. As the titrant is added, the first excess of silver ions forms a red silver chromate precipitate, marking the endpoint. The solution pH should be maintained at about 8 by adding solid CaCO3.
In the Volhard method, a standard excess of AgNO3 is first added to the...
In the Volhard method, a standard excess of AgNO3 is first added to the...
1.9K
01:12Precipitation Processes
495
The experimental conditions in a gravimetric analysis should be optimized to maximize the particle size and purity of the obtained precipitate. Ideally, the concentration of the precipitating reagent should be low with effective stirring to maintain low relative supersaturation for the growth of large crystals. In homogeneous precipitation, the precipitant is slowly generated by a chemical reaction in the solution to avoid local reagent excesses. For example, urea decomposes gradually to...
495
01:00Detection of Gross Error: The Q Test
6.3K
When one or more data points appear far from the rest of the data, there is a need to determine whether they are outliers and whether they should be eliminated from the data set to ensure an accurate representation of the measured value. In many cases, outliers arise from gross errors (or human errors) and do not accurately reflect the underlying phenomenon. In some cases, however, these apparent outliers reflect true phenomenological differences. In these cases, we can use statistical methods...
6.3K
01:02Quantifying and Rejecting Outliers: The Grubbs Test
1.7K
Sometimes, a data set can have a recorded numerical observation that greatly deviates from the rest of the data. Assuming that the data is normally distributed, a statistical method called the Grubbs test can be used to determine whether the observation is truly an outlier. To perform a two-tailed Grubbs test, first, calculate the absolute difference between the outlier and the mean. Then, calculate the ratio between this difference and the standard deviation of the sample. This...
1.7K