Discriminant Analysis under f-Divergence Measures.

Area of Science:

• Statistical inference
• Information theory
• Machine learning

Background:

• High-dimensional data poses computational challenges for statistical inference and performance limits.
• Dimensionality reduction methods can mitigate these challenges but often reduce performance.
• Preserving statistical divergence during dimensionality reduction is crucial for maintaining inference capabilities.

Purpose of the Study:

• To investigate linear dimensionality reduction techniques that maximally preserve statistical divergence between Gaussian models.
• To characterize the optimal linear transformation for discriminant analysis under various f-divergence measures.
• To explore the design of subspace projections for both zero-mean and general Gaussian models.

Main Methods:

• Linear dimensionality reduction
• Discriminant analysis
• Analysis of five f-divergence measures (Kullback-Leibler, symmetrized KL, Hellinger, total variation, chi-squared)
• Characterization for zero-mean Gaussian models
• Numerical algorithms for general Gaussian models

Main Results:

• Optimal linear projections for zero-mean Gaussian models do not always align with the largest covariance modes.
• In some cases, optimal projections align with the smallest covariance modes.
• A universal design for subspace projection emerges under specific conditions, independent of the chosen f-divergence measure.

Conclusions:

• Linear dimensionality reduction can be optimized to preserve statistical divergence in Gaussian models.
• The optimal projection strategy can be counter-intuitive, sometimes favoring smaller covariance modes.
• The universality of the optimal design across different divergence measures simplifies the selection of dimensionality reduction techniques.