Improving Moderator Responsiveness in Online Peer Support Through Automated Triage.

Area of Science:

• Digital mental health
• Online peer support systems
• Machine learning applications in healthcare

Background:

• Online peer support forums require careful moderation to ensure user safety and therapeutic value.
• Increasing community size burdens human moderators, risking overlooked individuals in need.
• Machine learning (ML) offers a potential solution to assist moderators by directing attention to critical content.

Purpose of the Study:

• To evaluate the accuracy of an automated machine learning triage system for online peer support messages.
• To assess the impact of this ML triage system on moderator behavior, specifically response ratios and latency.
• To determine if ML can effectively direct moderator attention in high-volume online mental health communities.

Main Methods:

• A machine learning classifier was developed to categorize messages (green, amber, red, crisis) based on urgency.
• The system was implemented on ReachOut.com, an Australian youth mental health platform.
• Moderator response ratios and latency were compared pre- and post-implementation, with and without formal training.

Main Results:

• The ML algorithm achieved an 84% f-measure for identifying content requiring moderator response.
• Response ratios increased for crisis, red, and green messages post-training, while decreasing for amber messages.
• Response latency was significantly reduced across all priority levels, particularly for crisis, red, and amber messages (77-80% reduction).

Conclusions:

• The ML triage system demonstrated good accuracy and aligned with moderator judgment.
• The system substantially reduced moderator response times, though its impact on response ratios was modest.
• Further research is needed to understand the effects of ML errors and improved responsiveness on user well-being.