LIT-LVM: Structured Regularization for Interaction Terms in Linear Predictors using Latent Variable Models.

Area of Science:

• Statistics
• Machine Learning
• Data Science

Background:

• Linear predictors are fundamental in statistics and machine learning.
• Modeling non-linear relationships often requires interaction terms, which can lead to high-dimensional challenges.
• Existing regularizers like lasso and elastic net help mitigate overfitting but may not fully capture complex interaction structures.

Purpose of the Study:

• To develop a method for accurate estimation of coefficients for interaction terms in linear predictors.
• To introduce a structured regularization approach based on a hypothesized low-dimensional structure of interaction coefficients.
• To provide interpretable low-dimensional latent representations of features.

Main Methods:

• Proposed a novel approach, LIT-LVM (Latent Interaction Terms - Latent Vector Model), which assumes interaction coefficients possess an approximate low-dimensional structure.
• Represented each feature by a latent vector in a low-dimensional space.
• Evaluated LIT-LVM against established methods like elastic net, hierarchical lasso, and factorization machines.

Main Results:

• LIT-LVM demonstrated superior prediction accuracy across diverse simulated and real-world datasets.
• The method showed particular effectiveness when the number of interaction terms is large relative to the number of samples.
• Achieved better performance compared to elastic net, hierarchical lasso, and factorization machines.

Conclusions:

• The hypothesized low-dimensional structure of interaction coefficients is effective for improving prediction accuracy.
• LIT-LVM offers a powerful structured regularization technique for high-dimensional data.
• The latent representations generated by LIT-LVM aid in feature visualization and relationship analysis.