Chatbot Language - crowdsource perceptions and reactions to dialogue systems to inform dialogue design decisions

Area of Science:

• Human-Computer Interaction
• Natural Language Processing
• Software Engineering

Background:

• Conversational User Interfaces (CUI) have a large global user base.
• Designing effective CUI requires careful selection of dialogue strategies.
• Evaluating CUI dialogue strategies through user testing is crucial but can be resource-intensive.

Purpose of the Study:

• To introduce Chatbot Language (CBL), a framework simplifying CUI development and deployment on crowdsourcing platforms.
• To enable researchers without technical backgrounds to conduct CUI user studies.
• To facilitate time- and cost-effective collection of user perceptions and reactions to CUI dialogue strategies.

Main Methods:

• Developed CBL, a JavaScript library with specialized CUI functionalities.
• Created scripts to interact with the Mechanical Turk (MT) API for creating Human Intelligence Tasks (HITs) and retrieving results.
• Utilized CBL to conduct experiments on the MT platform, demonstrating a sample workflow.

Main Results:

• CBL allows for rapid development and deployment of CUI on crowdsourcing platforms.
• Researchers can conduct CUI experiments on MT without requiring advanced technical skills.
• The framework streamlines the process of gathering user data for dialogue strategy evaluation.

Conclusions:

• CBL significantly lowers the barrier to entry for CUI research using crowdsourcing.
• The framework supports efficient and accessible user testing of dialogue strategies.
• CBL offers a valuable tool for the CUI research community, with potential for future enhancements.