Case reports unlocked: Leveraging retrieval-augmented generation with large language models to advance research on...

Research on psychological child maltreatment is impeded by a lack of high-quality structured data. Crucial information is often documented in child protective services (CPS) case files, but only in narrative form. Recent research on the utilization of retrieval-augmented generation (RAG) methods with large language models (LLMs) for extracting structured data from narratives has demonstrated significant potential. RAG methods can facilitate automated classification, thereby eliminating the need for laborious annotation.

We aimed to extract structured data from narrative casework reports by utilizing RAG and LLMs to classify mentions of 24 CPS case factors. These factors encompass child maltreatment indicators, risk factors associated with parental, family, and child characteristics, CPS interventions, and their outcomes. We focused on examining the extraction of psychological abuse due to its complex nature and difficulty in assessing this phenomenon. The results were compared with parental lack of cooperation, a factor with a presumed medium level of recognition difficulty, and a more straightforward factor of parental alcohol abuse.

We developed a four-stage workflow comprising of (1) case reports collection, (2) RAG based assessment of case factor mentions, (3) automated extraction of case factors from RAG assessments, and (4) case labeling. All CPS reports (N = 29,770) between 2008 and 2022 from Switzerland's largest CPS provider were collected. Model performance was evaluated compared against human-coded validation data on assessments. Two expert human reviewers independently classified weighted random samples of reports to validate the findings from which a consensus dataset was derived.

The model classified psychological abuse, lack of parental cooperation and parental alcohol abuse compared to a consensus dataset, with an accuracy of 82 %, 83 %, and 95 %, respectively, surpassing the agreement rates between the two human reviewers (79 %, 80 %, and 93 %).

RAG based assessment can replicate human judgment even on complex CPS case factors. High accuracy and complete inter-rater agreement level was achieved for factors that are straightforward to classify, such as parental alcohol abuse. The effectiveness of these methods stems from the presence of contextual clues related to case factors within a few sentences across different sections of the text, rather than from characteristics inherent to the entire text. For case factors such as parental lack of cooperation, both supporting and refuting evidence needs to be assessed to achieve optimal accuracy. Careful consideration of potential biases and limitations in RAG methods is advised. These applications can serve as early warning systems, by identifying critical factors from extensive case notes that might otherwise be overlooked, supporting professionals in making informed decisions and improving outcomes for at-risk children.