Random Partition Distribution Indexed by Pairwise Information | JoVE Visualize

Area of Science:

Statistics
Machine Learning
Computational Biology

Background:

Traditional clustering algorithms often use pairwise similarities, like distances, but integrating this into Bayesian models for prior partition distributions is complex.
Existing partition distributions may not effectively balance the probability of partitions with varying numbers of subsets.
Flexible Bayesian modeling requires prior distributions that can incorporate specific data characteristics, such as item similarities.

Purpose of the Study:

To develop a novel random partition distribution that is indexed by pairwise similarity information.
To create a flexible Bayesian modeling framework that utilizes pairwise similarities to define a prior partition distribution.
To ensure the proposed distribution allocates probability among partitions within a fixed number of subsets without shifting probability across different subset counts.

Main Methods:

A new random partition distribution is proposed, indexed by pairwise similarity information (distances).
The distribution is designed to allocate probability among partitions within a specified number of subsets.
The formulation includes a closed-form distribution for the number of subsets and a tractable normalizing constant for MCMC inference.

Main Results:

The proposed distribution assigns higher probability to partitions that group similar items together.
It maintains a constant total probability for partitions with a given number of subsets, regardless of similarity.
The distribution of the number of subsets and its moments are available in closed-form and are independent of the similarity measures.
The explicit probability mass function allows for the application of standard Markov Chain Monte Carlo (MCMC) methods for posterior inference.

Conclusions:

The novel random partition distribution offers attractive properties for Bayesian modeling, particularly in clustering applications.
It provides a principled way to incorporate pairwise similarities into prior partition distributions.
Demonstrations confirm the distribution's ability to highlight features and outperform existing methods in specific scenarios.