Development of a Novel Scoring System to Quantify the Severity of Incident Reports: An Exploratory Research Study

Incident reporting systems have been widely adopted to collect information about patient safety incidents. Much of the value of incident reports lies in the free-text section. Computer processing of semantic information may be helpful to analyze this. We developed a novel scoring system for decision making to assess the severity of incidents using the semantic characteristics of the text in incident reports, and compared its results with experts' opinions. We retrospectively analyzed free-text data from incident reports from January 2012 to September 2021 at Nagoya University Hospital, Aichi, Japan. The sample was allocated to training and validation datasets using the hold-out method. Morphological analysis was used to segment terms in the training dataset. We calculated a severity term score, a severity report score and severity group score, by report volume size, and compared these with conventional severity classifications by patient safety experts and reporters. We allocated 96,082 incident reports into two groups. We calculated 1,802 severity term scores from the 48,041 reports in the training dataset. There was a significant difference in severity report score between reports categorized as severe and not severe by experts (95% confidence interval [CI] -0.83 to -0.80, p < 0.001, d = 0.81). Severity group scores were positively associated with severity ratings from experts and reporters (correlation coefficients 0.73 [95% CI 0.63-0.80, p < 0.001] and 0.79 [95% CI 0.71-0.85, p < 0.001]) for all departments. Our severity scoring system could therefore contribute to better organizational patient safety.