Fast and powerful conditional randomization testing via distillation.

Area of Science:

• Statistics
• Machine Learning
• Bioinformatics

Background:

• Conditional independence testing is crucial for understanding variable relationships.
• Existing conditional randomization tests offer statistical validity but are computationally intensive.
• Integrating complex prediction algorithms into these tests is often infeasible due to computational cost.

Purpose of the Study:

• To develop a computationally efficient conditional randomization test.
• To leverage state-of-the-art machine learning algorithms within conditional randomization tests.
• To enable accurate and fast conditional independence testing, even with large datasets.

Main Methods:

• Proposed the distilled conditional randomization test.
• Introduced computational speed-up techniques like screening and recycling computations.
• Validated the approach using simulations and a real-world breast cancer dataset.

Main Results:

• The distilled conditional randomization test significantly reduces computational expense compared to existing methods.
• The proposed method maintains high statistical power and exact validity.
• Achieved orders of magnitude reduction in computation time, making it practical for large-scale applications.

Conclusions:

• The distilled conditional randomization test offers a practical solution for computationally intensive conditional independence testing.
• This approach effectively combines the power of machine learning with the statistical guarantees of conditional randomization tests.
• Demonstrated utility in identifying biomarkers related to cancer stage in a breast cancer dataset.