Evaluation of narrative text for case finding: the need for accuracy measurement

This article reviews the analysis of a narrative text electronic search technique being used in the insurance industry. We reviewed a previously published study of motor vehicle crashes in roadway construction workzones as well as additional data supplied by the authors with respect to the methods of keyword selection. The narrative text search technique was evaluated with decision statistics and was found to have a sensitivity of 92.3%, 95% confidence interval 67.5%-99.6%. This range of sensitivity, at its most extreme value, led to a 32.5% underestimation of claims prevalence. Furthermore, because the electronic search developed two classification categories from a limited text field (approximately 20 words), only half of the cases had at least one classification. Systematic error estimates were used to obtain true population proportions for crash characteristics, revealing significant underestimations in costs. This analysis highlights the need for investigators to apply decision statistics to narrative text searching techniques when they are used essentially as diagnostic test procedures on insurance claims datasets.