High throughput nonparametric probability density estimation

Area of Science:

• Statistical inference
• Computational statistics
• Data science

Background:

• High throughput applications in bioinformatics and finance require accurate probability distribution functions with limited data and no human subjectivity.
• Existing methods may struggle with complex data characteristics or require subjective parameter tuning.

Purpose of the Study:

• To develop an automated, objective method for estimating probability density functions (PDFs) for univariate continuous data.
• To provide a robust statistical inference tool applicable to high throughput applications.

Main Methods:

• Combines the maximum entropy method with single order statistics and maximum likelihood estimation.
• Employs a sample size invariant universal scoring function based on a quasi-log-likelihood function.
• Iteratively refines cumulative distribution functions, using the scoring function to identify and minimize atypical fluctuations.

Main Results:

• The developed method demonstrates resistance to under and over fitting, serving as an alternative to information criteria like AIC or BIC.
• Scaled quantile residual plots are introduced for effective visualization and quality assessment of PDF estimates.
• Benchmark tests confirm convergence to true PDFs for challenging distributions (discontinuities, heavy tails, singularities) as sample size increases.

Conclusions:

• The proposed method offers general applicability for high throughput statistical inference, particularly for complex and limited data scenarios.
• The technique provides robust probability density estimation without relying on prior distributional assumptions or subjective input.
• The method's accuracy and generalizability are validated across diverse and difficult test cases.