Validation of the AI/RHEUM knowledge base with data from consecutive rheumatological outpatients

The diagnostic accuracy of AI/RHEUM, an experimental expert system for support in the diagnosis of rheumatic diseases, was assessed using a collection of data in a cohort of 1,570 consecutive outpatients of a Dutch rheumatological clinic. Computer diagnoses based on these data and diagnostic predictions made by rheumatologists were compared with reference diagnoses that had been obtained by consensus of rheumatologists after 6-12 months follow-up. Performance of the tested version of the AI/RHEUM knowledge base is presented by various methods. Sensitivity varied between 29% and 100% for different rheumatological diseases. Average sensitivity and specificity for all 26 diagnoses present in the knowledge base were 67% and 98%, respectively. Performance according to the level of confidence indicated that 78% of the "definite", 65% of the "probable", and 33% of the "possible" conclusions made by AI/RHEUM were in agreement with the reference diagnoses. These results approximated the predictions made by rheumatologists after a single, initial examination. The system was less accurate than it had appeared in previous evaluation studies with complex clinical cases. The AI/RHEUM knowledge base needed refining to diagnose early rheumatic complaints. This study further illustrates the need for objective and informative parameters for expressing accuracy of diagnostic support systems.