A sparse negative binomial mixture model for clustering RNA-seq count data.

Area of Science:

• Bioinformatics
• Computational Biology
• Statistical Genetics

Background:

• Clustering small-n-large-p data is crucial for analyzing modern biological datasets.
• Current methods often rely on Gaussian assumptions, which are unsuitable for RNA sequencing count data.
• Normalization of count data can lead to loss of information and biased results.

Purpose of the Study:

• To develop a novel clustering method specifically designed for RNA sequencing count data.
• To address the limitations of existing methods that assume continuous data.
• To enable accurate sample clustering and simultaneous feature selection in high-dimensional transcriptomic studies.

Main Methods:

• Development of a negative binomial mixture model for count data clustering.
• Incorporation of lasso and fused lasso regularization for gene selection.
• Utilizing a modified Expectation-Maximization (EM) algorithm for model inference.
• Employing the Bayesian Information Criterion (BIC) for tuning parameter selection.

Main Results:

• The proposed negative binomial mixture model demonstrates superior clustering accuracy compared to existing methods.
• Effective feature selection of relevant genes is achieved through regularization techniques.
• The method provides enhanced biological interpretation of results, particularly in pathway analysis.
• Successful application to real-world transcriptomic data from rat brain and breast cancer studies.

Conclusions:

• The developed negative binomial mixture model is a powerful tool for clustering RNA sequencing count data.
• This approach overcomes the limitations of Gaussian-based methods for count data.
• The method facilitates robust sample classification, accurate gene selection, and meaningful biological insights in transcriptomics.