Evaluation of an ARF diagnosis calculator: a survey and content analysis

Acute Rheumatic Fever (ARF) is a critically important condition for which there is no diagnostic test. Diagnosis requires the use of a set of criteria comprising clinical, laboratory, electrocardiographic and echocardiographic findings. The complexity of the algorithm and the fact that clinicians lack familiarity with ARF, make ARF diagnosis ideally suited to an electronic decision support tool. The ARF Diagnosis Calculator was developed to assist clinicians in diagnosing ARF and correctly assign categories of 'possible, 'probable' or 'definite' ARF. This research aimed to evaluate the acceptability, accuracy, and test performance of the ARF Diagnosis Calculator.

Three strategies were used to provide triangulation of data. Users of the calculator employed at Top End Health Service, Northern Territory, Australia were invited to participate in an online survey, and clinicians with ARF expertise were invited to participate in semi-structured interviews. Qualitative data were analysed using inductive analysis. Performance of the calculator in correctly diagnosing ARF was assessed using clinical data from 35 patients presenting with suspected ARF. Diagnoses obtained from the calculator were compared using the Kappa statistic with those obtained from a panel of expert clinicians.

Survey responses were available from 23 Top End Health Service medical practitioners, and interview data were available from five expert clinicians. Using a 6-point Likert scale, participants highly recommended the ARF Diagnosis Calculator (median 6, IQR 1), found it easy to use (median 5, IQR 1) and believed the calculator helped them diagnose ARF (median 5, IQR 1). Clinicians with ARF expertise noted that electronic decision making is not a substitute for clinical experience. There was high agreement between the ARF Diagnosis Calculator and the 'gold standard' ARF diagnostic process (κ = 0.767, 95% CI: 0.568-0.967). Incorrect assignment of diagnosis occurred in 4/35 (11%) patients highlighting the greater accuracy of expert clinical input for ambiguous presentations. Sixteen changes were incorporated into a revised version of the calculator.

The ARF Diagnosis Calculator is an easy-to-use, accessible tool, but it does not replace clinical expertise. The calculator performed well amongst clinicians and is an acceptable tool for use within the clinical setting with a high level of accuracy in comparison to the gold standard diagnostic process. Effective resources to support clinicians are critically important for improving the quality of care of ARF.