Finding distributions that differ, with false discovery rate control

Area of Science:

• Statistics
• Machine Learning

Background:

• Comparing multiple distributions to a reference is crucial in data analysis.
• Existing methods may lack robustness or require distributional assumptions.

Purpose of the Study:

• To develop a novel methodology for identifying comparison groups with distributions differing from a reference group.
• To provide exact, distribution-free control of the false discovery rate (FDR) in multiple-testing scenarios.

Main Methods:

• Introduction of 'batch conformal p-values'.
• Demonstration of positive regression dependence across groups, enabling FDR control via the Benjamini-Hochberg procedure.
• Novel proof technique for rank vector construction under exchangeability.

Main Results:

• The proposed method achieves exact, distribution-free FDR control.
• Simulations show performance comparable to distribution-specific methods and superior power over direct conformal detection.
• Application to hepatitis C and Current Population Survey datasets identified significant patient and subpopulation groups.

Conclusions:

• Batch conformal p-values offer a powerful, distribution-free approach for multiple distribution comparison.
• The methodology is effective in identifying meaningful differences in real-world data.
• This work advances FDR control techniques in statistical inference.