Efficiently quantifying dependence in massive scientific datasets using InterDependence Scores.

Area of Science:

• Computational Biology
• Data Science
• Bioinformatics

Background:

• Modern scientific datasets are massive, featuring millions of samples and tens of thousands of variables.
• Existing dependence measures like Pearson correlation are limited to linear relationships and do not scale well.
• Discovering complex, nonlinear dependencies is crucial for novel insights in large-scale data.

Purpose of the Study:

• Introduce the InterDependence Score (IDS), a novel, scalable measure for quantifying both linear and nonlinear dependencies.
• Develop an efficient algorithm for IDS computation suitable for high-dimensional, large-scale datasets.
• Demonstrate IDS's utility in identifying key variables, topics, and biological relationships.

Main Methods:

• IDS is inspired by dependence measures in infinite-dimensional Hilbert spaces, capturing all dependence types.
• An efficient, linear-time algorithm leveraging neural network principles is employed for computation.
• The algorithm is optimized for parallel processing on GPUs, enabling analysis of billions of variable pairs.

Main Results:

• IDS successfully identifies relevant variables for predictive modeling tasks.
• The method effectively extracts word sets representing topics from large document corpora.
• IDS reveals gene sets associated with "gene-expression programs" in massive single-cell datasets.

Conclusions:

• IDS offers a scalable and effective solution for detecting diverse dependencies in large scientific datasets.
• Its speed and ability to capture nonlinear relationships make it a valuable tool for data exploration and insight generation.
• IDS has broad applicability across various scientific domains dealing with high-dimensional data.