Characterizing Documented Psychosocial Stressors in Pediatric Psychiatric Emergencies with an Open-Weight Large...

Area of Science:

• Psychiatry
• Artificial Intelligence
• Data Science

Background:

• Pediatric psychiatric intake notes contain valuable psychosocial information.
• Manual extraction of this data is time-consuming and resource-intensive.
• Large language models (LLMs) offer potential for automated data extraction.

Purpose of the Study:

• To evaluate the efficacy of a locally hosted open-weight LLM in extracting documented psychosocial factors from pediatric psychiatric intake notes.
• To assess the LLM's performance in a large emergency psychiatry cohort.
• To validate the LLM's extraction capabilities against human review.

Main Methods:

• Identified emergency department presentations for patients under 18 with psychiatric diagnoses.
• Utilized a locally hosted LLM (gpt-oss:120b) to classify psychosocial factors (peer conflict, sleep, school issues) from intake notes.
• Compared LLM-assisted human reliability with human-only reliability using Fleiss' kappa.
• Assessed model stability through repeated queries and applied the workflow to the full cohort.

Main Results:

• Human-plus-LLM reliability was comparable to human-only reliability (human kappa: 0.71-0.94; LLM-assisted kappa: 0.70-0.93).
• LLM-human agreement increased with model output stability.
• Commonly extracted factors included sleep disruption (57.7%), peer conflict (47.2%), and school-related issues (academic, disciplinary, attendance).

Conclusions:

• Locally hosted open-weight LLMs can reliably support structured extraction of psychosocial factors from pediatric psychiatric notes.
• Local validation is crucial for ensuring accurate and reliable data extraction.
• This approach enables scalable analysis of psychosocial factors in large cohorts.