External quality assessment in water microbiology: statistical analysis of performance

A UK-based scheme of water microbiology assessment requires participants to record counts of relevant organisms. Not every sample will contain the target number of organisms because of natural variation and therefore a range of results is acceptable. Results which are tail-end (i.e. at the extreme low or high end of this range) could occasionally be reported by any individual laboratory by chance. Several tail-end results might imply a laboratory problem. Statistical assessment is done in two stages. A non-parametric test of the distribution of tail-end counts amongst laboratories is performed (Cochran's Q) and, if they are not random, then observed and expected frequencies of tail-end counts are compared to identify participants who may have reported excessive numbers of low or high results. Analyses so far have shown that laboratories find high counts no more frequently than would be expected by chance, but that significant clusters of low counts can be detected among participants. These findings have been observed both in short-term and in long-term assessments, thus allowing detection of new episodes of poor performance and intermittent problems. The analysis relies on an objective definition of tail-end results. Working definitions are presented which should identify poor performance in terms of microbiological significance, and which allow fair comparison between membrane-filtration and multiple-tube techniques. Smaller differences between laboratories, which may be statistically significant, will not be detected. Different definitions of poor performance could be incorporated into future assessments.