Manifold-adaptive dimension estimation revisited.

Area of Science:

• Computational Neuroscience
• Machine Learning
• Signal Processing

Background:

• Data dimensionality is crucial for signal processing pipeline design.
• Nearest neighbor-based dimension estimators are vital for understanding data complexity.
• Existing estimators have limitations in accuracy and handling finite sample effects.

Purpose of the Study:

• To improve the manifold adaptive Farahmand-Szepesvári-Audibert (FSA) dimension estimator.
• To propose a robust global measure of intrinsic dimensionality using the median of local estimates.
• To evaluate the performance of the enhanced FSA estimator against other state-of-the-art methods.

Main Methods:

• Developed a probability density function for local FSA estimates under uniform manifold density.
• Derived a maximum likelihood formula for global intrinsic dimensionality under i.i.d. assumptions.
• Introduced an exponential correction formula to address edge and finite-sample effects, calibrated on hypercube datasets.

Main Results:

• The corrected median-FSA estimator demonstrates superior performance compared to the maximum likelihood estimator.
• It achieves comparable results to DANCo on standard synthetic benchmarks using mean percentage error and error rate metrics.
• The median-FSA algorithm revealed significant changes in neural dynamics during resting state and epileptic seizures.

Conclusions:

• The enhanced median-FSA estimator provides a more accurate and robust measure of intrinsic dimensionality.
• Identified brain areas with lower-dimensional neural dynamics as potential seizure onset zones.
• This method offers a valuable tool for analyzing complex neural data and understanding neurological disorders.