A mutual information criterion with applications to canonical correlation analysis and graphical models

Area of Science:

• Statistics
• Machine Learning
• Information Theory

Background:

• Model selection in statistical analysis often relies on criteria like Akaike's Information Criterion (AIC).
• Existing small-sample corrections for information criteria can be complex to apply.
• There is a need for more direct and accessible methods for variable and model selection.

Purpose of the Study:

• To derive a new criterion for deciding conditional independence.
• To develop a method that is easier to apply than existing small-sample corrections of AIC.
• To provide a criterion applicable to variable selection in canonical correlation analysis and graphical model selection.

Main Methods:

• Derivation of a novel criterion based on information theory principles.
• Demonstration of consistency with small-sample corrections of AIC.
• Establishing the criterion's reduction to mutual information under ideal distribution assumptions.

Main Results:

• Introduction of the Mutual Information Criterion (MIC).
• MIC is shown to be consistent with small-sample AIC corrections.
• MIC simplifies the process of variable selection in canonical correlation analysis and graphical model selection.

Conclusions:

• The Mutual Information Criterion (MIC) provides a more direct and applicable method for conditional independence decisions.
• MIC offers a valuable alternative to existing, less accessible, small-sample criteria.
• This criterion enhances statistical model selection, particularly in complex analyses like canonical correlation and graphical models.