A psychometric experiment in causal inference to estimate evidential weights used by epidemiologists

A psychometric experiment in causal inference was performed on 159 Australian and New Zealand epidemiologists. Subjects each decided whether to attribute causality to 12 summaries of evidence concerning a disease and a chemical exposure. The 1,748 unique summaries embodied predetermined distributions of 19 characteristics generated by computerized evidence simulation. Effects of characteristics of evidence on causal attribution were estimated from logistic regression, and interactions were identified from a regression tree analysis. Factors with the strongest influence on the odds of causal attribution were statistical significance (odds ratio = 4.5 if 0.001 < or = P < 0.05 and 7.2 if P < 0.001, vs P > or = 0.05); refutation of alternative explanations (odds ratio = 8.1 for no known confounder vs none adjusted); strength of association (odds ratio = 2.0 if 1.5 < relative risk < or = 2.0 and 3.6 if relative risk > 2.0, vs relative risk < or = 1.5); and adjunct information concerning biological, factual, and theoretical coherence. The refutation of confounding reduced the cutpoint in the regression tree for decision-making based on strength of association. The effect of the number of supportive studies reached saturation after it exceeded 12 studies. There was evidence of flawed logic in the responses concerning specificity of effects of exposure and a tendency to discount evidence if the P-value was a "near miss" (0.050 < P < 0.065). Evidential weights based on regression coefficients for causal criteria can be applied to actual scientific evidence.