Computerized comparison of six adverse drug reaction assessment procedures

Several standardized assessment procedures are currently used in the evaluation of adverse drug reactions (ADRs). Disagreement in rating ADRs can result from between-raters variability and between-methods differences in weighting the evidence. We eliminated between-raters variability by computer simulation of 1134 ADRs (including all the possible combinations of criteria currently used) and by automatic rating using different algorithms adapted from six published methods. Percentage agreement (Po) and weighted kappa test (kappa w) between pairs of methods are always better than with randomized scores, but the strength of agreement is only moderate (0.26 less than Po less than 0.59; 0.14 less than kappa w less than 0.51). The weightings of criteria are evaluated in terms of sensitivity, specificity, and predictive values. Criteria are neither sensitive (0.41 less than Se less than 0.70) nor specific (0.18 less than Sp less than 0.63) and have poor predictive values. Disagreements on weightings are considerable for three major criteria: timing of event, dechallenge, and alternative etiologic candidates. We discuss some ways of improving reliability of ADR diagnosis.