An intelligent agent for sentence completion test: creation and application in depression assessment

Area of Science:

• Psychological Assessment
• Artificial Intelligence in Mental Health
• Clinical Psychology

Background:

• Traditional self-report questionnaires suffer from response deception and social desirability bias.
• The Sentence Completion Test (SCT) is a projective technique with potential but faces challenges in manual scoring and high costs.
• Advancements in Large Language Models (LLMs) offer opportunities to overcome limitations in traditional psychological assessment methods.

Purpose of the Study:

• To develop and validate an intelligent agent using LLMs for depression assessment in Chinese university students based on the SCT framework.
• To evaluate the reliability, validity, and consistency of the LLM-powered SCT agent compared to manual scoring and established depression scales.
• To assess the agent's capability in identifying invalid responses during psychological screening.

Main Methods:

• Development of a specialized set of SCT items tailored for depression assessment.
• Integration of SCT's theoretical framework with LLM capabilities to create a self-built intelligent agent.
• Three progressive empirical studies involving Chinese university students, including reliability and validity analyses (Cronbach's alpha, exploratory factor analysis), criterion correlations (Beck Depression Inventory, Self-Rating Depression Scale), and invalid response detection metrics (F1, Accuracy, Precision, Recall).

Main Results:

• The intelligent agent demonstrated good reliability (Cronbach's α = 0.89-0.92) and validity.
• High consistency with manual SCT scoring (r = 0.96) and significant criterion correlations with the Beck Depression Inventory (r = 0.89) and Self-Rating Depression Scale (r = 0.85).
• The agent achieved high performance in identifying invalid responses (F1 = 0.94, Accuracy = 0.99, Precision = 0.99, Recall = 0.90).

Conclusions:

• This research successfully demonstrates the intelligent transformation of the Sentence Completion Test using Large Language Models for depression assessment.
• The LLM-powered agent offers a reliable, valid, and efficient alternative to traditional methods, overcoming limitations of manual scoring and potential biases.
• This innovation provides new academic and practical pathways for psychological assessment, particularly in large-scale screening contexts.