Validation of the medical expert system RENOIR

RENOIR is an expert system developed to assist the diagnosis of 37 diseases of connective tissue and inflammatory arthropathies. Precise diagnosis of rheumatic diseases implies great uncertainty and there is no gold standard with which to compare the expert system output. To overcome this problem a set of clinical cases was submitted to RENOIR and its diagnoses were compared with those of clinicians. Medical records of 81 patients with rheumatic diseases were interpreted by RENOIR and by 12 clinicians at three different expertise levels in rheumatology. Distances between the likelihoods of the 37 considered diseases provided by clinicians and RENOIR were computed as a disagreement measure. Mahalanobis distance was used to correct the collinearity between the possibilities of each pair of diseases. Using the resulting matrices of distances between experts, cluster analyses were carried out to classify RENOIR among human experts. Greater differences between RENOIR and clinicians than among clinicians themselves were not found.