Monitoring methods in the surgical arena: assumptions and potential pitfalls

Utilization of statistical graphical methods to detect deterioration and to compare performance of health providers (e.g., surgeons, surgical departments, centers, etc.) has been rapidly increasing. These methods rely heavily on assumptions that may not be applicable in all surgical scenarios. The results produced by those methods could have major potential impact on funding, court rulings, insurance rates, etc. Thus, if one wants to use these graphical methods, it is imperative that the methods produce highly reliable results, even when some of the assumptions that underlie such methods are violated. In this manuscript, we discuss some of the assumptions that underlie such methods. We examine the performance of these methods when some assumptions are violated by using simulations based on analyses of plausible data. Our results show that using current graphical methods to compare two or more health providers when the assumptions are not met could result in misleading conclusions. Hence, researchers should apply these types of graphical methods with appropriate care, and only after making sure that the underlying assumptions are valid or the methods are robust enough to those violations.