Evaluation of an expert system producing geometric solids as output

This paper reports the evaluation of an expert system whose output is a three-dimensional geometric solid. Evaluating such an output emphasizes the problems of establishing a comparison standard, and of identifying and classifying deviations from that standard. Our evaluation design used a panel of physicians for the first task and a separate panel of expert judges for the second. We found that multi-parameter or multi-dimensional expert system outputs, such as this one, may result in lower overall performance scores and increased variation in acceptability to different physicians. We surmise that these effects are a consequence of the higher number of factors which may be deemed unacceptable. The effects appear, however, to be equal for computer and human output. This evaluation design is thus applicable to other expert systems producing similarly complex output.