Forecasting school violence risk with incomplete interview data: an automated assessment approach

School violence risk prevention in the United States relies on manual assessments that are time-consuming and subjective. We developed a machine learning algorithm named Automated RIsk Assessment (ARIA), using natural language processing (NLP) to find linguistic patterns in standardized interview questions that can predict risk of aggression. Our goal was to evaluate the incremental change in performance with the addition of each question to simulate situations where interviews cannot be completed.

Students were interviewed with 2 14-question risk assessments, the Brief Rating of Aggression by Children and Adolescents (BRACHA) and the School Safety Scale (SSS), that encouraged open-ended answers to the interview questions. The reference standard was defined as the subject's likeliness to display aggression in the future as determined by a forensic psychiatrist. Feature sets were extracted to represent the addition of 1 question at a time in a typical interview, up to and including the 28 total main questions along with other sub-questions that arose. The ARIA NLP pipeline tokenized each feature set, then extracted n-gram features (n ≤ 5) that captured contextual and semantic information. The features were evaluated using an L2-regularized logistic regression classifier and L2-regularized support vector machine (L2-SVM) classifier.

Between May 1, 2015 and February 6, 2021, 412 assessment interviews were conducted. When compared to clinical judgement, ARIA performed with an area under the Receiver Operating Characteristic curve of 0.9 after 10 BRACHA questions, suggesting that it remains powerful even with truncated interviews. The full BRACHA had similar performance to the BRACHA + SSS assessment.

ARIA could use incomplete risk assessment interviews to provide modest recommendations even if interview completion is not possible. This could help to reduce the burden for the social worker or school counselor who may be using ARIA in less-than-ideal conditions.