Subject level clustering using a negative binomial model for small transcriptomic studies.

Area of Science:

• Bioinformatics
• Computational Biology
• Genomics

Background:

• Unsupervised clustering is key for analyzing high-throughput 'omics data.
• Existing methods often struggle with limited sample sizes and over-dispersed RNA-sequencing (RNA-seq) count data.
• The Negative Binomial distribution is frequently used to model over-dispersed RNA-seq data.

Purpose of the Study:

• To develop and evaluate a Negative Binomial Model-Based (NBMB) clustering approach for RNA-seq studies.
• To compare NBMB performance against Gaussian Mixture Modeling (GMM) in various simulation scenarios.
• To apply NBMB for sample clustering in real-world RNA-seq datasets.

Main Methods:

• Developed a Negative Binomial Model-Based (NBMB) clustering method.
• Employed a stochastic version of the expectation-maximization algorithm for clustering.
• Conducted simulation studies and applied the method to type 2 diabetes and ovarian cancer RNA-seq datasets.

Main Results:

• NBMB outperformed GMM in most scenarios with limited sample sizes.
• NBMB showed superior performance compared to GMM for small cluster distances, irrespective of sample size.
• Analysis of real RNA-seq data demonstrated good agreement between NBMB-identified subtypes and known disease classifications.

Conclusions:

• Negative Binomial model-based clustering is recommended for analyzing over-dispersed RNA-seq count data.
• The NBMB method provides a robust approach for sample clustering in RNA-seq studies.
• The NBMB method is available as an R package (NB.MClust) on CRAN.