Using artificial intelligence to support rapid, mixed-methods analysis: Developing an automated qualitative assistant...

Area of Science:

• Qualitative research methodology
• Computational social science
• Artificial intelligence in social sciences

Background:

• Qualitative data analysis is time-consuming and costly, limiting its application.
• Traditional AI methods like Latent Semantic Indexing/Latent Dirichlet Allocation (LSI/LDA) struggle with the nuances of qualitative data.
• There is a need for more effective AI tools to support qualitative research.

Purpose of the Study:

• To develop and evaluate an AI platform, the Automated Qualitative Assistant (AQUA), for augmenting qualitative data coding.
• To assess AQUA's performance against human coders using Cohen's kappa.
• To determine the efficiency of AQUA in terms of time and cost.

Main Methods:

• Developed AQUA using a graph-theoretic topic extraction and clustering approach, replacing LSI/LDA.
• Trained AQUA on a dataset with 11 qualitative categories and 72 subcategories, previously established by human coders.
• Compared AQUA's coding performance (using cosine-similarity) and time investment against human coders on free-text responses from 538 participants.

Main Results:

• AQUA achieved a low overall Cohen's kappa (~0.45) when coding all categories.
• For specific 3-category clusters with reduced linguistic diversity, AQUA demonstrated comparable inter-coder reliability (kappa 0.62-0.72) to human coders.
• AQUA reduced coding time from approximately 30 person hours to 5 hours (including human interpretation).

Conclusions:

• The Automated Qualitative Assistant (AQUA) can enhance the efficiency of qualitative data analysis.
• AQUA is effective in identifying categories suitable for automated coding, particularly those with less linguistic diversity.
• This AI-driven approach offers significant time and cost savings for qualitative researchers analyzing large datasets.