Using Prosodic and Lexical Information for Learning Utterance-level Behaviors in Psychotherapy.

Area of Science:

• Computational linguistics
• Psychology
• Artificial intelligence

Background:

• Psychotherapy outcome prediction is crucial for effective treatment.
• Analyzing session dynamics requires robust feature extraction from speech and text.
• Existing methods often overlook the contribution of prosodic and fine-grained lexical information.

Purpose of the Study:

• To develop and evaluate a novel approach for predicting utterance-level behaviors in psychotherapy sessions.
• To investigate the predictive power of prosodic and lexical features, both manually and automatically transcribed.
• To explore the utility of attention mechanisms in identifying key linguistic and acoustic patterns.

Main Methods:

• Utilized long short-term memory (LSTM) networks with an attention mechanism.
• Incorporated manually and automatically transcribed word-level lexical features.
• Integrated word-level prosodic features (e.g., pitch, energy, duration).
• Fused prosodic and lexical features for enhanced predictive modeling.

Main Results:

• Prosodic features were found to contain discriminative information for behavior prediction.
• Fusion of prosodic and automatically derived lexical features significantly improved prediction accuracy.
• Attention mechanism analysis highlighted important words and prosodic patterns for behavior prediction.

Conclusions:

• Speech and lexical features, particularly prosodic elements, are valuable for predicting psychotherapy behaviors.
• LSTM networks with attention offer a powerful framework for analyzing complex conversational data.
• This approach can lead to more objective and data-driven insights into therapeutic interactions.