Performance of the Self-Controlled Case Series for Drug Safety Signal Detection: A Multi-Database Study

Differences in performance of the Self-Controlled Case Series (SCCS) for signal detection have been reported across different databases. However, there has been limited comparative analysis of performance and it remains unknown whether combinations of databases could enable more effective signal detection.

This study aims to compare the performance of the SCCS for signal detection across several data sources, and to determine whether combinations of databases can improve SCCS performance.

We applied the SCCS to macrolides and fluoroquinolone antibiotics, in four databases: Merative MarketScan Commercial Claims and Medicare, the Clinical Practice Research Datalink (CPRD) Aurum and the Système National des Données de Santé. We developed a reference set of 104 positive controls and 58 negative controls, using a taxonomy framework to ensure the selected drug outcome pairs are theoretically well suited to the SCCS design. The observation period lasted 2 years, with a 30-day risk-window after each dispensing. Diagnostic performance was measured using sensitivity, specificity and area under the receiver operating curve (AUC) with respect to the product labels, both for individual and combinations of databases.

The sensitivity of the SCCS ranged from 0.57-0.89 across individual databases, and the specificity from 0.43-0.77 when limited to drug-outcome pairs sufficiently powered. The combination of all databases achieved the maximum sensitivity of 0.89 (0.41 specificity) for the full reference set, and a sensitivity of 1 (0.35 specificity) for drug outcome pairs with enough power. Whilst AUCs ranged from 0.66 to 0.71 across individual databases, the highest performing combination was CPRD plus MarketScan Commercial Claims (0.76 AUC).

Using a carefully designed reference set of drug-outcome pairs well suited to the study design, the SCCS performance varied substantially by database due to differences in population, reporting, healthcare and coding systems and prescribing patterns. Multi-database studies showed increased performance of SCCS for signal detection.