基于最小规范化协差决定因素和主要组件分析的方法,用于识别高维稀疏数据中的高杆点
Siti Zahariah1,2, Habshah Midi2,3
1Applied Statistics and Data Science Cluster, Universiti Kuala Lumpur Malaysian Institute of Information Technology (UniKL MIIT), Kuala Lumpur, Malaysia.
Journal of applied statistics
|October 30, 2023
概括
一种新的方法,RMD-MRCD-PCA,有效地识别高维数据中的高杆点. 它改进了现有的方法,特别是当变量数量超过200时.
科学领域:
- 统计 统计 统计 统计
- 数据挖掘 数据挖掘
- 机器学习 机器学习
背景情况:
- 识别高杆点 (HLP) 在统计分析中至关重要.
- 现有的强大的马哈拉诺比斯距离 (RMD) 方法,如RMD-MRCD,与高维数据作斗争.
- 增加的独立变量 (p) 会降低RMD-MRCD的性能.
研究的目的:
- 提出一种新的方法,RMD-MRCD-PCA,用于在高维稀疏数据中识别HLPs.
- 随着维度的增加,解决RMD-MRCD的性能限制.
- 提高HLP识别技术的稳定性和适用性.
主要方法:
- 通过将主要组件分析 (PCA) 集成到最小规则化协差决定器 (MRCD) 算法中,开发了RMD-MRCD-PCA.
- PCA组件缩小了共变矩阵,确保了RMD计算的可逆性.
- 利用模拟研究和两个真实数据集进行比较分析.
主要成果:
- 对于p ≈ 200,RMD-MRCD-PCA表现出与RMD-MRCD可比的性能.
- 在p>200时,RMD-MRCD性能显著下降,特别是在p=700时.
- 坚固的PCA (ROBPCA) 显示无效,由于沼泽问题造成的污染率低于20%.
结论:
- RMD-MRCD-PCA为高维数据集中的HLP识别提供了一个强大的解决方案.
- 拟议的方法克服了传统的RMD-MRCD的局限性,随着维度的增长.
- 在某些污染场景中,RMD-MRCD-PCA为ROBPCA提供了更可靠的替代方案.
相关概念视频
Principal Moments of Area
1.1K
In mechanics, the product of inertia and moments of inertia of area help to calculate the stability and performance of various structures and components. The coordinate transformation relations are used to calculate the moments and products of inertia for an area about the inclined axes. Further, the moments and products of inertia with respect to the principal axes can be determined using the moments and products of inertia about the inclined axes.
The principal moment of inertia axes are the...
The principal moment of inertia axes are the...
1.1K
Vector Algebra: Method of Components
13.9K
It is cumbersome to find the magnitudes of vectors using the parallelogram rule or using the graphical method to perform mathematical operations like addition, subtraction, and multiplication. There are two ways to circumvent this algebraic complexity. One way is to draw the vectors to scale, as in navigation, and read approximate vector lengths and angles (directions) from the graphs. The other way is to use the method of components.
In many applications, the magnitudes and directions of...
In many applications, the magnitudes and directions of...
13.9K
Outliers and Influential Points
4.1K
An outlier is an observation of data that does not fit the rest of the data. It is sometimes called an extreme value. When you graph an outlier, it will appear not to fit the pattern of the graph. Some outliers are due to mistakes (for example, writing down 50 instead of 500), while others may indicate that something unusual is happening. Outliers are present far from the least squares line in the vertical direction. They have large "errors," where the "error" or residual is the...
4.1K
Residuals and Least-Squares Property
7.4K
The vertical distance between the actual value of y and the estimated value of y. In other words, it measures the vertical distance between the actual data point and the predicted point on the line
If the observed data point lies above the line, the residual is positive, and the line underestimates the actual data value for y. If the observed data point lies below the line, the residual is negative, and the line overestimates the actual data value for y.
The process of fitting the best-fit...
If the observed data point lies above the line, the residual is positive, and the line underestimates the actual data value for y. If the observed data point lies below the line, the residual is negative, and the line overestimates the actual data value for y.
The process of fitting the best-fit...
7.4K
Extraction: Partition and Distribution Coefficients
2.5K
The distribution law or Nernst's distribution law is the law that governs the distribution of a solute between two immiscible solvents. This law, also known as the partition law, states that if a solute is added to the mixture of two immiscible solvents at a constant temperature, the solute is distributed between the two solvents in such a way that the ratio of solute concentrations in the solvents remains constant at equilibrium.
For extracting a solute from an aqueous phase into an...
For extracting a solute from an aqueous phase into an...
2.5K
Calibration Curves: Linear Least Squares
1.3K
A calibration curve is a plot of the instrument's response against a series of known concentrations of a substance. This curve is used to set the instrument response levels, using the substance and its concentrations as standards. Alternatively, or additionally, an equation is fitted to the calibration curve plot and subsequently used to calculate the unknown concentrations of other samples reliably.
For data that follow a straight line, the standard method for fitting is the linear...
For data that follow a straight line, the standard method for fitting is the linear...
1.3K


