Testing high-dimensional multinomials with applications to text analysis

Area of Science:

• Multivariate statistics
• Computational statistics
• Machine learning

Background:

• Comparing discrete probability distributions is vital for text mining, topic modeling, and authorship attribution.
• Existing methods often require assumptions about parameter homogeneity or equal sample sizes, limiting their applicability.
• High-dimensional multinomial distributions present unique challenges due to the curse of dimensionality.

Purpose of the Study:

• To develop a novel statistical test for the equality of probability mass functions across K groups of high-dimensional multinomial distributions.
• To establish the asymptotic properties of the proposed test statistic under the null hypothesis.
• To demonstrate the test's optimality and practical utility in real-world scenarios.

Main Methods:

• A new test statistic is proposed for comparing multinomial probability mass functions.
• Asymptotic null distribution of the test statistic is derived as standard normal.
• The test's ability to achieve the optimal detection boundary is theoretically established.
• Simulation studies and real-world dataset analyses are conducted.

Main Results:

• The proposed test statistic has an asymptotic standard normal distribution under the null hypothesis.
• The limiting null distribution is parameter-free and does not require equal group sizes or identical parameters within groups.
• The test achieves the optimal detection boundary across the parameter space.
• Simulations confirm the test's performance, and applications show its utility in analyzing customer reviews and scientific abstracts.

Conclusions:

• A robust and asymptotically optimal test for comparing high-dimensional multinomial distributions is presented.
• The method offers a powerful tool for applications in text mining, topic modeling, and discrete distribution analysis.
• The test's parameter-free null distribution and optimality make it broadly applicable without stringent assumptions.