Precision Verification: Effect of Experiment Design on False Acceptance and False Rejection Rates

We examined the false acceptance rate (FAR) and false rejection rate (FRR) of varying precision verification experimental designs.

Analysis of variance was applied to derive the subcomponents of imprecision (ie, repeatability, between-run, between-day imprecision) for complex matrix experimental designs (day × run × replicate; day × run). For simple nonmatrix designs (1 day × multiple replicates or multiday × 1 replicate), ordinary standard deviations were calculated. The FAR and FRR in these different scenarios were estimated.

The FRR increased as more samples were included in the precision experiment. The application of an upper verification limit, which seeks to cap FRR at 5% for multiple experiments, significantly increased the FAR. The FRR decreases as the observed imprecision increases relative to the claimed imprecision and when a greater number of days, runs, or replicates are included in the verification design. Increasing the number or days, runs, or replicates also reduces the FAR for between-day imprecision and repeatability.

Design of verification experiments should incorporate the local availability of resources and analytical expertise. The largest imprecision component should be targeted with a greater number of measurements. Consideration of both FAR and FRR should be given when committing a platform into service.