Evaluation of Epilepsy Expert--a decision support system

Epilepsy Expert is a decision support system based on the International Classification of Epilepsies and Epileptic Syndromes (1989). The aim of this study was to evaluate the Epilepsy Expert. First the diagnostic performance was validated. This was done in 3 stages: collection of the patient cases, determination of the 'correct diagnoses' and testing the system. How the users perceived the functionality of the system was studied by using an inquiry. Three physicians, experts of epilepsy, from different hospitals were asked to choose 10 patients. In the patient description was a short history, a detailed description of the seizure, EEG findings and their own diagnosis. Next, each expert made a diagnosis of the cases supplied by other experts by using the International Classification of Epilepsies and Epileptic Syndromes. The 'correct diagnosis' (so-called majority agreement) was the diagnosis given by the majority of the experts. The diagnosis of each expert was compared with the 'correct diagnosis'. The diagnoses obtained by the Epilepsy Expert were then compared with the 'correct diagnoses'. In the evaluation the expert physicians agreed on 37% of cases and all 3 disagreed on 17%. A majority agreed on 25 cases, which were used in the evaluation. In these 25 cases the experts' (A,B,C) diagnoses were correct or partly correct in 100, 64, 80% of cases, respectively. The program's diagnoses were correct or partly correct in 80% (module I) and 76% (modules IV and V) of cases. In the evaluation Epilepsy Expert was found to be only partly successful. The main reason for this was the weakness of the international classification. However, the program seems to be very close to the level of the experts. According to this limited inquiry Epilepsy Expert is not suitable for clinical use, because it is, for example, too simple and does not contain enough information.