Prediction in road safety studies: an empirical inquiry

Studies about the road safety effect of interventions are usually retrospective quasi-experiments. In these, one key task is to predict what would have been the safety of the treated group without the intervention. Such predictions can be made by several methods, one of which is to use a "comparison group." We use 26 yearly counts of reported injury accidents for the Canadian provinces to examine which of several simple methods of prediction would have historically predicted best. We find that the use of more data does not always improve prediction. How well one predicts depends not only on the amount of data used but also on the extent to which the prediction method is in accord with the unknown time trend behind the accident counts. We also find that the use of a comparison group to predict is not always better than predicting that this year's count will be the same as last year's. In addition, the intuitive notion that a good comparison group is that which is thought similar to the treated group is too simple. Both similarity and size (as measured by the number of accidents) are important. Moreover, whatever preconceived notions of similarity we had, were contradicted by the data. If the history of accident counts on the treatment group and on several possible comparison groups is available, a simple method to select the most suitable comparison group is suggested.