A Case Study on Assessing AI Assistant Competence in Narrative Interviews

Area of Science:

• Social Sciences
• Artificial Intelligence
• Human-Computer Interaction

Background:

• AI's role is expanding beyond data analysis to social research interaction.
• The impact of AI on narrative interviews, a collaborative data collection method, remains underexplored.
• Effective narrative interviewing requires interviewer skills like empathy and structured questioning.

Purpose of the Study:

• To investigate the potential of AI in conducting narrative interviews.
• To evaluate AI's performance in maintaining interview structure, empathy, and narrative quality.
• To assess the utility of observation-based metrics for non-technical researchers evaluating AI-driven interviews.

Main Methods:

• A case study using an OpenAI Assistant on WhatsApp to conduct narrative interviews.
• A participant shared a story twice: once standard, once deliberately deviating.
• AI performance evaluated via conversation analysis and narrative indicators (structure, empathy, coherence, agency support).

Main Results:

• The AI demonstrated adaptability and structure maintenance in conversations.
• Findings illustrate AI's potential for personalized and flexible narrative interviews.
• The study successfully tested metrics for evaluating AI-driven narrative interviews.

Conclusions:

• Observation-based metrics can help non-technical social researchers assess AI interview quality.
• The study prompts reflection on AI's evolving role in qualitative social research.
• Results encourage further research into AI applications for narrative data collection.