Variable Selection for Sparse Data with Applications to Vaginal Microbiome and Gene Expression Data.

Area of Science:

• Statistics
• Bioinformatics
• Genomics

Background:

• Sparse data, characterized by a high proportion of zeros, is prevalent across scientific disciplines.
• Analyzing and modeling sparse, high-dimensional data presents significant statistical and computational challenges.
• Existing methods may not adequately address the complexities of real-world sparse datasets.

Purpose of the Study:

• To develop and present robust statistical methods and tools for analyzing sparse, high-dimensional data.
• To illustrate the application of these methods using real-world scientific datasets.
• To provide a framework for identifying significant features and improving predictive modeling in sparse data contexts.

Main Methods:

• Utilized zero-inflated model selection and significance testing for comparative analysis.
• Applied techniques to identify significant time intervals in longitudinal microbiome data.
• Employed feature selection methods to identify a subset of informative genes from a larger dataset.

Main Results:

• Successfully identified significant differences in *Lactobacillus* species between pregnant and non-pregnant groups.
• Selected the top 50 genes from 2426 sparse gene expression data, achieving 100% prediction accuracy.
• Demonstrated that the first four principal components of selected genes explain 83% of the model variability.

Conclusions:

• The proposed statistical methods are effective for analyzing complex sparse, high-dimensional data.
• The techniques facilitate the identification of biologically relevant features and improve classification accuracy.
• This approach offers a valuable tool for microbiome and gene expression data analysis.