DiSCoKit: An open-source toolkit for deploying live LLM experiences in survey research

Area of Science:

• Human-Computer Interaction
• Artificial Intelligence
• Social Sciences

Background:

• Studying human-AI interaction dynamics requires presenting participants with live large-language model (LLM) stimuli.
• Technical and practical challenges, including survey platform limitations and data logging, hinder the deployment of AI stimuli in online surveys.
• Existing methods often lack the flexibility to manipulate AI behaviors for experimental designs.

Purpose of the Study:

• To introduce DiSCoKit, an open-source toolkit designed to facilitate the deployment of live LLM experiences within JavaScript-enabled survey platforms.
• To provide researchers with a solution that overcomes common technical barriers in survey-based AI stimulus deployment.
• To enable naturalistic LLM interactions for experimental research in human-AI dynamics.

Main Methods:

• Developed DiSCoKit, an open-source toolkit compatible with JavaScript-enabled survey platforms like Qualtrics.
• Integrated LLM models, such as those available through Microsoft Azure, for live interaction.
• Focused on addressing challenges in survey platform integration, chat data logging, and AI behavior manipulation for experimental control.

Main Results:

• DiSCoKit enables the deployment of live LLM stimulus experiences through online surveys.
• The toolkit offers a flexible, secure, and scalable solution for researchers.
• Demonstrated toolkit deployment and customization with an example, highlighting its possibilities and limitations.

Conclusions:

• DiSCoKit provides a practical and effective solution for researchers investigating human-AI interaction dynamics.
• The toolkit democratizes the use of live LLM stimuli in experimental research.
• Facilitates more sophisticated and naturalistic studies of human-AI collaboration and interaction through online surveys.