AI as the Interpreter for Identifying Root Causes and Emotional Themes in Mental Health Narratives on Reddit Using...

Mental health issues continue to increase worldwide, often intensified by stigma and lack of awareness. Social media has become a major space where individuals articulate their emotional and psychological experiences. However, little is known about how AI models interpret these narratives, particularly when contextual reasoning and alignment with human judgment are required.

This study examines 2 AI models, AutoML and PaLM 2, to classify mental health root causes in social media posts and evaluate their alignment with human judgment. The aim is to assess AI-human agreement and model interpretability when applied to complex, real-world mental health narratives.

AutoML and PaLM 2 were trained to classify mental health root causes using an annotated dataset of Reddit (Reddit, Inc.) posts (n=800). Model generalizability and alignment with human judgment were evaluated on a hold-out set of unseen posts (n=50) using human consensus labels. Quantitative analyses included AI-human agreement assessment and statistical comparison of paired predictions. Qualitative analyses examined misclassification patterns, contextual reasoning, and interpretability using a structured coding approach.

On the unseen posts, PaLM 2 aligned with human consensus on 40/50 (80%) posts, while AutoML aligned on 34/50 (68%) posts. Agreement between PaLM 2 and human raters (Cohen κ≈0.72) fell within a similar range to that observed for AutoML (Cohen κ≈0.67). In the paired comparison of model errors, the McNemar test based on discordant pairs (PaLM 2 correct and AutoML incorrect: 12; AutoML correct and PaLM 2 incorrect: 6) did not indicate a difference in classification outcomes between models (χ²₁=1.39, P=.24). Qualitative analysis identified differences in the output patterns of the models when processing contextual and emotional information, with PaLM 2 often generating outputs consistent with broader narrative context and AutoML relying more heavily on lexical features.

AutoML showed strong internal performance yet reduced generalizability to unseen mental health narratives. PaLM 2 demonstrated distinct alignment patterns relative to human consensus and outputs consistent with greater sensitivity to narrative context, although both models did not differ significantly in error distribution. Both models and human raters struggled with inherently ambiguous categories, underscoring the complexity of mental health discourse. These findings highlight the value of combining quantitative and interpretability-focused analysis to advance transparent AI for mental health text classification.