Choosing thresholds for statistical signal detection with the proportional reporting ratio

Identification of potential drug safety problems using statistical screening algorithms in routinely collected databases of adverse drug reactions (ADRs) requires decision rules based on thresholds of the chosen parameters. Choosing higher or lower thresholds changes both the sensitivity of the screening and the number of false alarms produced, and thus has an impact on the effectiveness of the detection process.

The aim of this study was to evaluate the impact on the effectiveness of signal detection activities of choosing different warning thresholds for the proportional reporting ratio (PRR) and for the count of reports of any drug-event combination.

Signal detection methods were tested within the EudraVigilance database of suspected ADRs. Using an established set of known ADRs, the number that could be detected and the changes in time gained for earlier investigation of the signal were calculated over a range of signal detection thresholds. These figures were set against the number of false positive signals produced by the statistical signal detection algorithms.

Higher thresholds for the lower confidence bound of the PRR produced fewer false positives but this benefit was offset by important losses of sensitivity in the detection of ADRs. By contrast, increases in the threshold for the count of a specific drug-event combination produced fewer false positives with little loss of either sensitivity or time gained for investigation of adverse events. A threshold of five compared with the current European Medicines Agency threshold of three gave a reduction of 25 % in false positive signals in return for a loss of 12 % in true signals detected early.

Changes in the standard threshold for the count of drug-event combinations can result in a substantial improvement in efficiency of the signal detection process. Initially this change might be applied only to products with a well-established safety profile.