Improved visualization of high-dimensional data using the distance-of-distance transformation

Area of Science:

• Data Science
• Computational Neuroscience
• Machine Learning

Background:

• Dimensionality reduction techniques like t-SNE and UMAP are crucial for analyzing high-dimensional biological and machine learning data.
• A common challenge is the 'scattering noise problem,' where random noise points obscure underlying data structures in low-dimensional embeddings.
• This issue hinders accurate interpretation of neuronal population activity and cellular genetic profiles.

Purpose of the Study:

• To develop an improved dimensionality reduction method capable of handling datasets with scattered noise points.
• To enhance the clarity and interpretability of low-dimensional embeddings derived from complex, noisy high-dimensional data.
• To demonstrate the efficacy of the proposed technique in biological and machine learning contexts.

Main Methods:

• A novel transformation of the distance matrix is proposed, focusing on the distance between neighbor distances.
• This transformation is applied to high-dimensional datasets, including neuronal spike sequences and convolutional neural network representations.
• The method aims to isolate noise points into a distinct cluster within the low-dimensional embedding.

Main Results:

• The proposed technique effectively alleviates the 'scattering noise problem' by separating noise points from genuine data clusters.
• Application to neuronal spike data revealed improved visualization of spiking patterns.
• Analysis of natural image representations by CNNs showed enhanced cluster separation and noise identification.

Conclusions:

• The presented method offers a significant improvement over existing dimensionality reduction techniques for noisy high-dimensional data.
• This approach enhances the reliability of data analysis in fields relying on accurate visualization of complex datasets.
• The technique provides a robust solution for distinguishing signal from noise in applications ranging from neuroscience to artificial intelligence.