Exploring the application of large language models in coding the experiencing scale (EXP).

Area of Science:

• Psychology
• Artificial Intelligence
• Computational Linguistics

Background:

• Psychotherapy process measures, such as the Experiencing Scale (EXP), are crucial for understanding clinical interactions.
• Manual coding of these measures is time-consuming and resource-intensive.
• Large language models (LLMs) present a potential solution for automating this coding process.

Purpose of the Study:

• To explore the feasibility of using ChatGPT for coding the Experiencing Scale (EXP).
• To provide a proof-of-concept for LLM-assisted psychotherapy process coding.
• To investigate ethical considerations, limitations, and future research directions for LLM application in this domain.

Main Methods:

• ChatGPT was employed to code 79 psychotherapy transcripts from the EXP manual.
• Multiple ChatGPT models and few-shot learning prompt engineering protocols were tested.
• Modal and peak EXP scores were rated across three data collection phases.

Main Results:

• ChatGPT demonstrated moderate agreement with manual reference ratings for EXP scores.
• An efficient configuration (o3-mini, 5-shot prompting) achieved moderate reliability for both modal (ICC=.67) and peak (ICC=.71) EXP scores.
• Preliminary evidence suggests LLMs can potentially augment or replace human coders under specific circumstances.

Conclusions:

• LLMs show promise for streamlining the coding of psychotherapy process measures like the EXP.
• Current evidence is preliminary, with remaining ethical and technical limitations.
• Future research should focus on out-of-manual data validation, pretraining exposure assessment, and privacy-preserving LLM applications.