Kernel embeddings and the separation of measure phenomenon

Area of Science:

• Statistics
• Machine Learning
• Information Theory

Background:

• Nonparametric two-sample testing is crucial for comparing probability distributions.
• Kernel methods are widely used in machine learning but their theoretical underpinnings for distribution comparison are complex.
• Distinguishing between continuous probability distributions can be challenging, especially in high dimensions.

Purpose of the Study:

• To demonstrate that kernel covariance embeddings provide information-theoretically perfect separation of distinct continuous probability distributions.
• To establish an equivalence between testing for the equality of probability measures and testing for the singularity of Gaussian measures in a reproducing kernel Hilbert space.
• To elucidate the theoretical mechanism behind the effectiveness of kernel methods in statistical inference.

Main Methods:

• Utilizing kernel covariance embeddings to map probability distributions to Gaussian measures in a reproducing kernel Hilbert space.
• Leveraging the Feldman-Hájek dichotomy for statistical proofs.
• Analyzing the properties of Gaussian measures in infinite-dimensional Hilbert spaces.

Main Results:

• Kernel covariance embeddings achieve information-theoretically perfect separation of distinct continuous probability distributions.
• Testing the equality of two nonatomic probability measures is equivalent to testing the singularity of their corresponding Gaussian embeddings.
• A small perturbation in a continuous distribution is maximally magnified by its Gaussian embedding, revealing a "separation of measure phenomenon".

Conclusions:

• Kernel covariance embeddings offer a powerful and theoretically sound approach for distinguishing between probability distributions.
• This method simplifies complex statistical testing problems by transforming them into more manageable Gaussian singularity tests.
• The findings provide a mathematical foundation for the success of kernel methods in various inference tasks.