Pearson's goodness-of-fit tests for sparse distributions

Area of Science:

• Statistics
• Probability Theory
• Data Analysis

Background:

• Pearson's chi-squared test is a standard method for assessing goodness-of-fit for categorical data against discrete distributions.
• Traditional theory assumes a fixed number of categories (k), where the test statistic converges to a chi-squared distribution as sample size (n) increases.
• Real-world scenarios often involve a number of categories that varies with sample size, potentially exceeding it.

Purpose of the Study:

• To investigate the asymptotic behavior of Pearson's chi-squared test statistic when the number of categories is not fixed.
• To develop a more powerful statistical test for goodness-of-fit in situations with a large or variable number of categories.
• To provide a robust methodology applicable to real-world datasets, such as lottery data.

Main Methods:

• Application of martingale techniques to analyze the convergence properties of the chi-squared test statistic.
• Development and proposal of a novel test statistic designed for improved power.
• Simulation studies to compare the performance of the proposed statistic against the traditional chi-squared statistic.
• Illustrative application using real-world lottery data.

Main Results:

• Proof that Pearson's chi-squared test statistic converges to a normal distribution under more general conditions, including when k is not fixed.
• Simulation results indicate the proposed new test statistic demonstrates superior power compared to the standard chi-squared test.
• The methodology is validated through a practical case study involving lottery data analysis.

Conclusions:

• The study extends the theoretical understanding of Pearson's chi-squared test, showing its convergence to a normal distribution under broader conditions.
• A novel, more powerful test statistic is introduced, offering an improved alternative for goodness-of-fit testing.
• The findings and proposed methods are practically relevant, as demonstrated by the lottery data application.