Related Experiment Video
Updated: Jan 1, 2026
13:44
Detection of Architectural Distortion in Prior Mammograms via Analysis of Oriented Patterns
Published on: August 30, 2013
43.5K
Detection of Outliers in Projection-Based Modeling
Oxana Ye Rodionova1, Alexey L Pomerantsev1
1N. N. Semenov Federal Research Center for Chemical Physics , RAS , Kosygin str. 4 , 119991 Moscow , Russia.
Analytical Chemistry
|December 28, 2019
Summary
A new method uses total distance in projection modeling for outlier detection in regression. This data-driven approach effectively identifies and removes outliers, improving model accuracy.
Area of Science:
- Chemometrics
- Data Science
- Statistical Modeling
Background:
- Principal Component Analysis (PCA) is established for data exploration and classification.
- Regression modeling requires robust methods for handling outliers that can distort results.
Purpose of the Study:
- To develop a novel outlier detection method for regression problems using projection modeling.
- To introduce a sequential procedure that addresses masking and swamping effects of outliers.
Main Methods:
- Calculation of total distance for each sample in projection models.
- Data-driven estimation of degrees of freedom and scaling parameters for outlier thresholding.
- Sequential outlier detection procedure with iterative and backward steps.
Main Results:
- Demonstrated effectiveness of the total distance approach in identifying outliers in regression.
- Illustrated the procedure's ability to overcome masking and swamping effects.
- Validated the method on pharmaceutical and agricultural (whole wheat spectra) datasets.
Conclusions:
- The developed sequential outlier detection procedure is effective for regression analysis.
- This method enhances the reliability of projection models by accurately handling outliers.
- The approach is applicable to diverse scientific datasets, including chemical and spectral data.
Related Concept Videos
01:08Outliers and Influential Points
5.8K
An outlier is an observation of data that does not fit the rest of the data. It is sometimes called an extreme value. When you graph an outlier, it will appear not to fit the pattern of the graph. Some outliers are due to mistakes (for example, writing down 50 instead of 500), while others may indicate that something unusual is happening. Outliers are present far from the least squares line in the vertical direction. They have large "errors," where the "error" or residual is the...
5.8K
01:12What Are Outliers?
4.9K
Outliers are observed data points that are far from the least squares line. They have unusual values and need to be examined carefully. Though an outlier may result from erroneous data, at other times, it may hold valuable information about the population under study and should be included in the data. Hence, it is crucial to examine what causes a data point to be an outlier.
The z score is used to find outliers or unusual values. It should be noted that any values beyond -2 and +2 are...
The z score is used to find outliers or unusual values. It should be noted that any values beyond -2 and +2 are...
4.9K
01:02Quantifying and Rejecting Outliers: The Grubbs Test
3.4K
Sometimes, a data set can have a recorded numerical observation that greatly deviates from the rest of the data. Assuming that the data is normally distributed, a statistical method called the Grubbs test can be used to determine whether the observation is truly an outlier. To perform a two-tailed Grubbs test, first, calculate the absolute difference between the outlier and the mean. Then, calculate the ratio between this difference and the standard deviation of the sample. This...
3.4K
01:00Detection of Gross Error: The Q Test
6.8K
When one or more data points appear far from the rest of the data, there is a need to determine whether they are outliers and whether they should be eliminated from the data set to ensure an accurate representation of the measured value. In many cases, outliers arise from gross errors (or human errors) and do not accurately reflect the underlying phenomenon. In some cases, however, these apparent outliers reflect true phenomenological differences. In these cases, we can use statistical methods...
6.8K
00:57Modified Boxplots
10.8K
A standard box and whisker plot informs us about the spread of the data in a given sample. One can identify the minimum value, maximum value, first quartile value, second quartile or median value, and third quartile.
However, the box plot does not tell the reader about outliers - values that lie far from the center of the data. We can modify the standard box and whisker plot to identify the outliers and visualize the actual spread of the data in a sample.
Initially, we calculate the adjusted...
However, the box plot does not tell the reader about outliers - values that lie far from the center of the data. We can modify the standard box and whisker plot to identify the outliers and visualize the actual spread of the data in a sample.
Initially, we calculate the adjusted...
10.8K
01:11Residuals and Least-Squares Property
8.8K
The vertical distance between the actual value of y and the estimated value of y. In other words, it measures the vertical distance between the actual data point and the predicted point on the line
If the observed data point lies above the line, the residual is positive, and the line underestimates the actual data value for y. If the observed data point lies below the line, the residual is negative, and the line overestimates the actual data value for y.
The process of fitting the best-fit...
If the observed data point lies above the line, the residual is positive, and the line underestimates the actual data value for y. If the observed data point lies below the line, the residual is negative, and the line overestimates the actual data value for y.
The process of fitting the best-fit...
8.8K