The AI interviewer: multi-faceted evaluation of adaptive questioning by large language models.

Area of Science:

• Artificial Intelligence
• Psychology
• Human-Computer Interaction

Background:

• Large language models (LLMs) are increasingly used as interviewers in research and human-computer interaction.
• Systematic evaluations of LLM interviewing behavior are limited, hindering their effective deployment.

Purpose of the Study:

• To introduce a modular LLM agent for semi-structured psychological interviews.
• To present a controlled, multi-faceted evaluation protocol for assessing interviewer quality across six state-of-the-art LLMs.

Main Methods:

• Developed an LLM agent for adaptive interviews with 54 main questions.
• Standardized interview context using human transcripts and a single LLM interviewee for fair comparison.
• Expert psycholinguists evaluated interviewer behavior on benevolence, necessity, context-awareness, openness, and justified skip, complemented by efficiency and linguistic metrics.

Main Results:

• Gemini 2.5 Pro demonstrated the most empathic tone; GPT-5 Chat optimized for speed and precision.
• Grok 4 achieved exhaustive coverage but with higher latency; Claude Sonnet 4 offered balanced versatility.
• Linguistic markers correlated with human judgments, indicating alignment between stylistic choices and perceived interview quality.

Conclusions:

• Systematic trade-offs exist in LLM interviewing capabilities, with distinct models excelling in different aspects.
• The developed toolkit enables principled deployment and auditing of LLM interviewers in psychological research.
• Researchers can now better match LLM capabilities to study goals for enhanced empathy, appropriateness, and effectiveness.