Eliciting principles of hazard identification from experts

National experts in the field of developmental toxicology were interviewed in order to elicit the principles, or rules-of-thumb, they use in determining if a compound or agent is likely to be a developmental hazard during pregnancy. Several levels of individual and cumulative consensus activity were carried out that resulted in consensus in 71 rules and partial consensus in an additional 24 rules of 145 rules initially elicited. Rules could be divided generically into those affecting the expert's confidence in a piece of scientific evidence and those determining the weight of importance of that evidence in deciding about hazard identification. Topically, the rules also divided into those about the general nature or characteristics of a compound, animal studies testing for an effect of the compound, and human reports about the presence of absence of developmental effects associated with the compound. Several conclusions about the methodology include the following: 1) expert systems must be based on the knowledge of more than one expert; 2) considerable human effort is expended in evaluating the certainty of scientific evidence before combining the evidence for problem solving; 3) how experts use evidence of different degrees of uncertainty in their decisions is a major area that is yet to be determined and that may greatly affect subsequent efforts in artificial intelligence; and 4) knowledge elicitation by interview has limitations but is a workable methodology for medical decision making.