Count-dependent filter for smoothing bivariate FCM histograms

A data-smoothing filter has been developed that permits the improvement in accuracy of individual elements of a bivariate flow cytometry (FCM) histogram by making use of data from adjacent elements, a knowledge of the two-dimensional measurement system point spread function (PSF), and the local count density. For FCM data, the PSF is assumed to be a set of two-dimensional Gaussian functions with a constant coefficient of variation for each axis. A set of space variant smoothing kernels are developed from the basic PSF by adjusting the orthogonal standard deviations of each Gaussian smoothing kernel according to the local count density. This adjustment in kernel size matches the degree of smoothing to the local reliability of the data. When the count density is high, a small kernel is sufficient. When the density is low, however, a broader kernel should be used. The local count density is taken from a region defined by the measurement PSF. The smoothing algorithm permits the reduction in statistical fluctuations present in bivariate FCM histograms due to the low count densities often encountered in some elements. This reduction in high-frequency spatial noise aids in the visual interpretation of the data. Additionally, by making more efficient use of smaller samples, systematic errors due to system drift may be minimized.