Developing an Automated Assessment of In-session Patient Activation for Psychological Therapy: Codevelopment Approach

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Health Informatics

Background:

• Patient activation, a key predictor of health outcomes and costs, is difficult to measure in healthcare consultations.
• Current methods for evaluating psychological therapy content are often time-consuming and costly.
• Natural Language Processing (NLP) offers a potential solution for accessible, systematic evaluation of therapy content.

Purpose of the Study:

• To apply responsible design principles in developing an AI model using NLP to automate patient activation ratings.
• To enhance the Consultation Interactions Coding Scheme (CICS) for assessing patient activation in psychological therapy.

Main Methods:

• Utilized 128 transcribed sessions of remotely delivered cognitive behavioral therapy from 53 participants with multiple health conditions.
• Employed participatory methodology with a multidisciplinary team, including service-user researchers and AI ethics experts, to identify key language features.
• Applied machine learning algorithms (k-nearest neighbors and bagged trees) to classify patient activation levels from transcribed therapy interactions.

Main Results:

• The k-nearest neighbors classifier achieved 73% accuracy, 82% precision, and 80% recall.
• The bagged trees classifier demonstrated 81% accuracy, 87% precision, and 75% recall in differentiating high, low, and neutral patient activation levels.

Conclusions:

• Collaboratively developed language features can effectively discriminate patient activation levels within psychological therapy sessions.
• This approach enables more accessible and systematic evaluation of patient activation for individuals with multiple long-term health conditions.