Supervised Natural Language Processing Classification of Violent Death Narratives: Development and Assessment of a...

Area of Science:

• Computational epidemiology
• Public health informatics
• Natural Language Processing (NLP)

Background:

• The National Violent Death Reporting System (NVDRS) now includes comprehensive law enforcement and medical examiner reports, offering vast potential for NLP research on violence.
• Unstructured narrative data within these reports presents a unique opportunity for advanced analytical techniques.

Purpose of the Study:

• To evaluate the application of supervised NLP, specifically using a compact large language model (LLM), to predict the circumstances and types of violent deaths.
• To assess the impact of data preprocessing, volume, and composition on NLP model performance within the NVDRS.

Main Methods:

• Applied distilBERT, a compact LLM, to unstructured narrative data from the NVDRS.
• Simulated the effects of preprocessing and training data characteristics on model performance, measuring F1-scores, precision, recall, and false negative rates.
• Assessed model bias across racial, ethnic, and sex subgroups by comparing F1-scores.

Main Results:

• A minimum of 1500 cases was required for a compact LLM to achieve an F1-score of 0.6 and a false negative rate of 0.01-0.05.
• Replacing domain-specific jargon significantly improved model performance.
• Oversampling to address class imbalance did not substantially enhance F1-scores.
• Observed F1-score disparities between racial/ethnic groups (0.2-0.25) and between sexes (0.12-0.2).

Conclusions:

• Compact LLMs are viable for supervised NLP tasks on imbalanced NVDRS data, provided sufficient training data.
• Simulations highlight the importance of preprocessing and training data strategies for LLM-based NLP applications.
• While effective, the models exhibited performance disparities across demographic groups, indicating a need for further bias mitigation research.