Low Bias Local Intrinsic Dimension Estimation from Expected Simplex Skewness.

Area of Science:

• Data Science
• Machine Learning
• Dimensionality Reduction

Background:

• Local intrinsic dimension estimation aids in discriminating between datasets from different low-dimensional structures.
• Adapting global estimators for local estimation often results in high bias or variance.

Purpose of the Study:

• To introduce a new method for local intrinsic dimension estimation.
• To address limitations of existing methods regarding bias and variance in high-dimensional data analysis.

Main Methods:

• Developed a novel method leveraging the 'curse/blessing of dimensionality'.
• Designed estimators with low bias and variance, effective even when intrinsic dimension exceeds data points.

Main Results:

• The proposed estimators demonstrate very low bias and relatively low variance.
• Achieved superior classification of local data sets by dimension compared to existing methods.

Conclusions:

• The new method offers a powerful tool for local intrinsic dimension estimation.
• Demonstrated practical utility in stratifying real-world datasets and classifying local data structures.