Agreement of expert judgment in causality assessment of adverse drug reactions

Global introspection is, with operational algorithms and Bayes' theorem, one of the three main approaches used to assess the causal relationship between a drug treatment and the occurrence of an adverse event.

To analyze and compare the judgments of five senior experts using global introspection about drug causation on a random set of putative adverse drug reactions.

A random sample of 150 drug-effect pairs was constituted. For each pair, five senior experts had to independently assess the probability of drug causation from 0 to 1 by using a 100 mm visual analog scale (VAS). For analysis, those probabilities were secondarily split into seven levels of causality: excluded (0-0.05); unlikely (0.06-0.25); doubtful (0.26-0.45); unassessable/unclassifiable (0.46-0.55); plausible (0.56-0.75); likely (0.75-0.95); and certain (0.95-1). Agreement among the five experts was assessed using kappa coefficients (kappa).

The overall agreement between experts was poor (kappa=0.20), although significantly different from chance, and varied according to the level of causality. It was lower for the unlikely, doubtful, unassessable/unclassifiable, and plausible categories (kappa=0.03, 0.03, -0.01, and 0.13, respectively) than for VAS extremes: excluded, likely, and certain (kappa=0.40, 0.32, and 0.30, respectively).

This study confirms that experts express marked disagreements when assessing drug causality independently. The agreement rate was lower for intermediate levels of causality, especially when strong evidence was lacking for confirming or ruling out drug causality. Therefore, in a decision-making context, a step-by-step consensual approach such as the Delphi method seems necessary to make the assessment of such cases more reliable.