Using Instruction-Tuned Large Language Models to Identify Indicators of Vulnerability in Police Incident Narratives.

Area of Science:

• Computational Social Science
• Artificial Intelligence in Criminology

Background:

• Police narrative reports offer insights into interactions with vulnerable populations.
• Qualitative coding of these reports is resource-intensive.

Purpose of the Study:

• To evaluate large language models (LLMs) for replicating human qualitative coding of police narrative reports.
• To assess LLM effectiveness in identifying specific vulnerabilities: mental ill health, substance misuse, alcohol dependence, and homelessness.

Main Methods:

• Comparison of human-generated and LLM-generated labels on Boston Police Department narrative reports.
• Assessment of various LLM sizes and prompting strategies.
• Analysis of label variability and counterfactual experiments for bias detection (sex, race).

Main Results:

• LLMs show high agreement with human coders, especially for non-vulnerable cases.
• Larger models and tailored prompts improve human-LLM agreement, with variation by vulnerability type.
• Counterfactual analyses revealed minimal bias related to sex and race in LLM classifications.

Conclusions:

• LLMs can significantly reduce manual coding requirements for narrative datasets.
• A human-LLM collaborative approach enhances coding specificity, transparency, and replicability.
• LLM application presents opportunities for criminology and related fields.