Validation of a Small Language Model for DSM-5 Substance Category Classification in Child Welfare Records

Area of Science:

• Natural Language Processing
• Machine Learning in Social Services
• Child Welfare Research

Background:

• Large language models (LLMs) show promise in classifying child welfare narratives for issues like substance use.
• Previous research focused on binary classification; classifying specific substance types with smaller, local models remained untested.

Purpose of the Study:

• To validate a locally hosted LLM for identifying specific substance types within child welfare investigation narratives.
• Aligning classification with DSM-5 substance categories.

Main Methods:

• A 20-billion-parameter LLM was used on child maltreatment narratives from a Midwestern U.S. state.
• A two-stage classification process identified seven DSM-5 substance categories.
• Human expert review assessed classification accuracy (precision, recall, Cohen's kappa) on 900 cases; reproducibility was tested on 15,000 records.

Main Results:

• Five substance categories (alcohol, cannabis, opioid, stimulant, sedative/hypnotic/anxiolytic) achieved high agreement (κ=0.94-1.00) with expert review.
• Classification precision for these categories ranged from 92% to 100%.
• High run-to-run agreement (92.1%-99.1%) was observed across categories, though low-prevalence categories performed poorly.

Conclusions:

• A small, locally hosted LLM reliably classifies specific substance types in child welfare text, advancing beyond binary detection.
• The pipeline operates on local hardware without altering data collection, enabling substance-specific surveillance and trend analysis.
• This technology can improve service alignment with the substance use profiles of investigated families.