Knowledge graphs based on meta-analysis papers improve the quality of case formulation: a mixed methods design

Area of Science:

• Psychology
• Artificial Intelligence
• Clinical Practice

Background:

• Case formulation (CF) is a crucial therapeutic skill, but its development is time-consuming.
• Existing methods for CF quality improvement are limited.
• The integration of AI and meta-analytic data offers a novel approach to enhance CF.

Purpose of the Study:

• To evaluate the effectiveness of knowledge graphs in improving the quality of therapist case formulations.
• To compare AI-generated CFs with human expert-generated CFs.
• To assess the impact of personalization prompts on AI-driven CF generation.

Main Methods:

• Five groups generated 25 case formulations (CFs) each from vignettes: Control (LLM), Personalization (LLM with prompts), Knowledge Graph (LLM with KG), Knowledge Graph with Personalization, and Human Expert.
• CFs were evaluated for correctness, completeness, feasibility, and consistency using a 7-point scale and binary scoring.
• Qualitative analysis examined the naturalness of language for client comprehension.

Main Results:

• Knowledge Graph and Knowledge Graph with Personalization groups showed significantly higher correctness, completeness, and feasibility than the control group.
• The Expert group achieved higher consistency scores than all machine-generated groups.
• No significant difference in feasibility was observed between Knowledge Graph, Knowledge Graph with Personalization, and Expert groups.
• Qualitative analysis indicated human CFs are more client-friendly in language.

Conclusions:

• Knowledge graphs enhance novice therapists' CF correctness, completeness, and feasibility.
• This AI-assisted method shows promise for improving the quality of mental health services.
• Further research may explore optimizing AI-generated CFs for natural client communication.