Scikit-Dimension: A Python Package for Intrinsic Dimension Estimation

Area of Science:

• Machine Learning
• Data Science
• Computational Statistics

Background:

• Accurate intrinsic dimensionality (ID) estimation is vital for managing uncertainty in real-life machine learning applications.
• Existing methods for ID estimation lack a standardized, user-friendly Python implementation.
• This gap hinders the consistent application and comparison of different ID estimation techniques.

Purpose of the Study:

• Introduce scikit-dimension, an open-source Python package for intrinsic dimension estimation.
• Provide a uniform interface for applying various ID estimation methods.
• Facilitate benchmarking of ID estimation techniques on diverse datasets.

Main Methods:

• Developed scikit-dimension leveraging the scikit-learn API for consistent implementation of ID estimators.
• Integrated generators for synthetic toy and benchmark datasets.
• Ensured code quality through testing, coverage, and continuous integration.

Main Results:

• Scikit-dimension offers a comprehensive toolkit for evaluating global and local intrinsic dimensions.
• The package facilitates large-scale benchmarking, analyzing over 500 datasets.
• Demonstrated the package's utility in comparing ID estimation methods on real-world and synthetic data.

Conclusions:

• Scikit-dimension addresses the need for a standardized Python package for intrinsic dimension estimation.
• The package simplifies the application and comparison of ID estimation methods.
• Enables robust evaluation of machine learning model performance by understanding data dimensionality.