Data segmentation based on the local intrinsic dimension.

Area of Science:

• Machine Learning
• Data Science
• Computational Statistics

Background:

• Machine learning often assumes data can be described by a few variables, defining the intrinsic dimension (ID).
• However, the ID can vary within a single dataset, a fact often overlooked in large-scale data analysis.
• Exploiting local variations in ID can provide deeper insights into data structure.

Purpose of the Study:

• To develop a robust and computationally efficient method for identifying and segmenting regions with different local intrinsic dimensions (ID) in high-dimensional data.
• To demonstrate the utility of local ID for unsupervised data segmentation across diverse real-world datasets.

Main Methods:

• Developed a novel, computationally efficient algorithm to discriminate regions with varying local intrinsic dimensions (ID).
• Applied the method to segment data points based on their local ID.
• Validated the approach on large, real-world datasets from various domains.

Main Results:

• Demonstrated that many real-world datasets exhibit significant heterogeneity in local intrinsic dimensions.
• Identified distinct data characteristics associated with different local IDs, including protein folding states, brain activity patterns, and financial risk profiles.
• Showcased the method's effectiveness in segmenting complex data, revealing underlying structural differences.

Conclusions:

• Local intrinsic dimension is a powerful topological feature for unsupervised data segmentation.
• This approach offers a complementary perspective to traditional clustering methods for analyzing high-dimensional data.
• The method provides valuable insights into the heterogeneous nature of real-world datasets.