Accuracy and reliability of assessment of severity of illness before and after an educational intervention

Severity-of-illness measurement is considered to be an important factor in the risk adjustment of medical outcomes. However, for those measures that involve a high level of clinical judgment, strategies have to be developed to maximize the consistency of rating severity that can be implemented, especially when rating is based upon medical record review. A group of 25 clinicians were sent the same set of 14 patient records, and requested to use the Duke University Severity of Illness (DUSOI) checklist to rate the severity of patients' illness. Written instructions for the use of this instrument were provided. A short educational intervention was than made by the research team, and the clinicians were sent the same set of records to be rated again. Any improvement in the accuracy and reliability of severity assessment over the two ratings was then studied. The educational intervention resulted in identification of fewer irrelevant health problems. However, it had little impact upon the reliability of severity scoring itself, as measured by the intraclass correlation coefficient. Measuring severity of illness is a conceptually complex procedure. However, given its role in outcome interpretation, it may be worth pursuing strategies aimed at maximizing consistency of clinician rating. There are a number of options, including improved written instructions and more intensive training, that could be implemented.