Biwhitening Reveals the Rank of a Count Matrix

Area of Science:

• Data Science
• Statistical Learning
• Matrix Analysis

Background:

• Estimating matrix rank is vital for Principal Component Analysis (PCA) and data analysis.
• Traditional methods struggle with count data matrices (e.g., Poisson) due to heteroskedastic noise.
• Random matrix theory offers tools but requires adaptation for count data.

Purpose of the Study:

• To develop a robust method for estimating the rank of Poisson random matrices with unknown variance.
• To address the challenge of heteroskedastic noise in count data for rank estimation.
• To provide a generalizable approach for various count distributions and missing data.

Main Methods:

• Proposed a biwhitening procedure to scale data matrices.
• Ensured noise spectrum aligns with the Marchenko-Pastur (MP) law for rank selection.
• Utilized the Sinkhorn-Knopp algorithm to estimate scaling factors from observations.

Main Results:

• Biwhitening effectively estimates the rank of Poisson parameter matrices without prior knowledge.
• The method extends to other distributions with quadratic mean-variance relationships (e.g., binomial, gamma).
• Numerical experiments and real-world datasets (scRNA-seq, Hi-C, topic modeling) validate the approach.

Conclusions:

• Biwhitening offers a powerful and versatile tool for rank estimation in corrupted count data matrices.
• The method demonstrates superior performance in challenging scenarios and various applications.
• This work advances the analysis of count-based data in diverse scientific fields.