Overview of agreement statistics for medical devices

This paper is an overview that summarizes recently developed tools in assessing agreement for methods comparison and instrument/assay validation in medical devices. This paper emphasizes concept, sample sizes, and examples more than analytical formulas. We have considered a unified approach of evaluating agreement among multiple instruments (k), each with multiple replicates (m) for both continuous and categorical data. We start with the basic scenario of two instruments (k = 2), each with only one measurement (m = 1). In this basic scenario for continuous data, we also consider if the target values are considered random (values of a gold standard instrument) or fixed (known values). In the more general case, we will not consider when the target values are fixed. We discuss the simplified sample size calculations. When there is a disagreement between methods, one needs to know if the source of the disagreement was due to a systematic shift (bias) or random error. The coefficients of accuracy and precision will be discussed to characterize these sources. This is important because a systematic shift usually can be easily fixed through calibration, while a random error usually is a more cumbersome variation reduction exercise. For categorical variables, we consider scaled agreement statistics. For continuous variables, we use scaled or unscaled agreement statistics. For variables with proportional error, we can simply apply a log transformation to the data. Finally, three examples are given: one for assay validation, one for a lab proficiency assessment, and one for a lab comparison on categorical assay.