A positive spin: large language models can help directors evaluate programs through their patients' own words

Area of Science:

• Pain Psychology
• Program Evaluation
• Qualitative Data Analysis

Background:

• Qualitative feedback is crucial for program evaluation, offering insights into patient experiences often missed by quantitative data alone.
• Relying solely on quantitative metrics risks overlooking the nuances of patient-reported outcomes and lived experiences.

Purpose of the Study:

• To assess the feasibility of using large language models (LLMs) for analyzing qualitative participant feedback in a pain education program.
• To explore the utility of LLM-assisted analysis in uncovering patient-centered themes for program improvement.

Main Methods:

• A dual-method approach combining LLM-assisted analysis (ChatGPT) and manual thematic review was employed.
• Qualitative responses from 82 participants of the Empowered Relief skill-based pain education class were analyzed.

Main Results:

• LLM analysis using ChatGPT identified 7 major themes: Use of Specific Audiofile, Mindset, Technique, Community and Space, Knowledge, Tools and Approaches, and Self-awareness.
• Thematic analysis yielded rich, unexpected insights directly from patients' own words, guiding program evaluation.

Conclusions:

• LLM-derived analysis provides an efficient and ergonomic method for extracting valuable qualitative insights, complementing traditional evaluation methods.
• This approach enables program directors to evaluate treatment outcomes more broadly, incorporating patient perspectives beyond disability measures.