Evaluating black-boxes as medical decision aids: issues arising from a study of neural networks

The rigorous evaluation of medical decision aids will be critical to promoting their development, establishing their clinical value and legalizing their use. Many decision aids are transparent in the sense that their internal structure and function can be examined and verified. Some decision aids, however, use complex models of associations in training data to construct 'black-box' systems whose workings are largely impenetrable and inexplicable. The issues surrounding the evaluation of such systems, as exemplified by connectionist (neural network) models, are discussed. For such systems the two major aspects that can be evaluated are the training data from which the system is derived, and its performance on test data. A number of questions about the use of black-box systems as medical decision aids are posed which require consideration by the medical informatics community.