Analyzing 10 years of early awareness and alert activity in the United kingdom

The aim of this study was to assess the accuracy of the English National Horizon Scanning Centre (NHSC) in identifying and filtering pharmaceutical developments using end user and international collaborator databases of emerging technologies as proxies for new drugs of likely significance to health services and/or patients.

We used the NHSC information system and the list of National Institute for Health and Clinical Excellence (NICE) technology appraisals to estimate the false positive rate for NHSC identification, filtration, and reporting. We assessed the sensitivity of NHSC identification and filtration of pharmaceuticals for NICE technology appraisals from 1999 to the end of December 2010, and for pharmaceuticals entered into the EuroScan International Network database.

We estimate that overall NHSC identification, filtration and reporting had a positive predictive value of 0.39 (95 percent CI, 0.36 to 0.43) and a false positive rate of 60 percent. Using NICE appraisals and EuroScan's database as proxies for pharmaceuticals of significance, we estimate the NHSC sensitivity over the 10-year period at 0.92 (95 percent CI, 0.89 to 0.95) and 0.89 (95 percent CI, 0.82 to 0.96) respectively.

Our results suggest that the NHSC has performed well in terms of sensitivity over the past decade, but that the false positive rate of 60 percent may indicate that the filtration criteria for pharmaceuticals could be tightened for increased efficiency. Future evaluations of EAA systems should include an element of external review and explore the level of accuracy acceptable to funders and customers of such systems.