Approximating mutual information of high-dimensional variables using learned representations

Area of Science:

• Computational Biology
• Information Theory
• Machine Learning

Background:

• Mutual information (MI) quantifies statistical dependence but is difficult to estimate in high dimensions due to sample size requirements.
• Existing MI estimation methods fail for datasets with more than tens of dimensions, limiting their application in complex biological systems.

Purpose of the Study:

• To develop a novel method for approximating MI in high-dimensional data by leveraging underlying low-dimensional structure.
• To demonstrate the effectiveness of the proposed latent MI (LMI) approximation method on benchmark datasets and real-world biological problems.

Main Methods:

• Developed latent MI (LMI) approximation, a method applying nonparametric MI estimation to low-dimensional data representations.
• Utilized a theoretically-motivated model architecture to learn low-dimensional representations from high-dimensional data.
• Validated LMI on benchmark datasets, comparing its performance against existing MI estimation techniques.

Main Results:

• LMI accurately approximates MI for variables with over 1000 dimensions, provided the data exhibits low intrinsic dimensionality.
• Demonstrated LMI's capability to analyze complex biological data, including protein language model representations and single-cell RNA sequencing data.
• Identified non-trivial information encoded by protein language models about protein-protein interactions.

Conclusions:

• Latent MI (LMI) approximation offers a scalable solution for estimating mutual information in high-dimensional biological datasets.
• The method successfully quantifies information in protein representations and reveals dynamic changes in cell fate information during hematopoietic stem cell differentiation.