An AI-powered research assistant in the lab: A practical guide for text analysis through iterative collaboration with...

Area of Science:

• Computational Social Science
• Natural Language Processing
• Artificial Intelligence

Background:

• Manual text analysis is time-consuming, labor-intensive, and prone to bias.
• Large Language Models (LLMs) offer a promising alternative for efficient and high-quality text analysis.
• Developing data-driven (bottom-up) taxonomies with LLMs requires structured methodologies.

Purpose of the Study:

• To present a step-by-step tutorial for developing, testing, and applying taxonomies using LLMs.
• To demonstrate an iterative and collaborative process between researchers and LLMs for taxonomy creation.
• To showcase the potential of LLMs in generating reliable bottom-up categorizations for unstructured data.

Main Methods:

• Utilized personal goals as example data for taxonomy development.
• Employed prompt engineering to guide LLM in reviewing datasets and generating an initial taxonomy.
• Iteratively refined the taxonomy through prompt modifications and direct researcher input.
• Applied the finalized taxonomy to categorize a complete dataset, assessing intercoder reliability.

Main Results:

• Achieved high human-LLM intercoder agreement in categorizing the dataset.
• Reduced text analysis time by approximately 87.5% compared to traditional methods.
• Demonstrated high intercoder reliability using the LLM-generated taxonomy.

Conclusions:

• LLMs can be effectively utilized to generate reliable bottom-up categorizations for unstructured text data.
• The presented iterative, collaborative approach facilitates efficient taxonomy development and application.
• This methodology offers a scalable solution for complex text analysis tasks, reducing bias and improving efficiency.