Automating the assessment of multicultural orientation through machine learning and natural language processing

Area of Science:

• Psychotherapy research
• Clinical psychology
• Artificial intelligence in mental health

Background:

• Multicultural Orientation (MCO) is crucial for effective psychotherapy.
• The MCO-performance task (MCO-PT) measures MCO capacities but requires manual coding.
• Automating MCO-PT scoring could enhance therapist training.

Purpose of the Study:

• To evaluate the feasibility of automating MCO-PT scoring using machine learning (ML) and natural language processing (NLP).
• To assess the reliability of ML-based scoring compared to human coders.

Main Methods:

• Utilized ML and NLP techniques to analyze transcripts of 613 MCO-PT responses from 100 participants.
• Compared automated scores against human-coded scores across MCO domains: cultural humility, opportunities, and comfort.

Main Results:

• ML models achieved near-human reliability for overall MCO (ρ = .75) and cultural opportunities (ρ = .81).
• Performance was less robust for cultural humility (ρ = .46) and cultural comfort (ρ = .41).

Conclusions:

• Automated scoring of MCO-PT using ML is feasible and shows high reliability for certain domains.
• This technology may enable the development of ML-based training tools for therapists to practice MCO skills.
• Further refinement is needed to improve accuracy for cultural humility and comfort aspects of MCO.