Outlier modeling for spectral data reduction.

Area of Science:

• Data science
• Spectroscopy
• Image processing

Background:

• Spectral reflectance datasets often contain correlated spectra.
• Principal Component Analysis (PCA) is used for lossy compression but is sensitive to outliers.
• Outlier spectra can significantly increase reconstruction errors.

Purpose of the Study:

• To introduce a novel outlier modeling (OM) method for spectral data compression.
• To improve the accuracy of spectral reconstruction in the presence of outliers.
• To enhance the performance of PCA by effectively handling outlier spectra.

Main Methods:

• Defined outliers using robust Mahalanobis distance with the fast minimum covariance determinant algorithm.
• Detected, clustered, and separately modeled outliers using cluster-specific PCA-derived bases.
• Applied PCA to the main dataset after outlier removal and modeled outliers independently.

Main Results:

• OM significantly improved PCA performance on the non-outlier data.
• Clustering outliers allowed for their separate modeling, preventing data loss.
• Tests demonstrated lower spectral reconstruction errors (normalized RMS and goodness of fit) using OM.

Conclusions:

• The proposed outlier modeling (OM) method effectively handles outlier spectra in spectral datasets.
• OM leads to more accurate spectral reconstruction compared to standard PCA.
• This method offers a robust strategy for compressing spectral reflectance data.