Exploring a GPT-based large language model for variable autonomy in a VR-based human-robot teaming simulation

Area of Science:

• Robotics
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Autonomous systems and robots are increasingly prevalent in digital environments.
• Integrating Large Language Models (LLMs) into human-robot teams offers potential for variable autonomy via verbal communication.

Purpose of the Study:

• To introduce a novel simulation framework for a GPT-powered multi-robot testbed.
• To explore user interaction and strategies within a simulated natural language-based multi-robot environment.
• To evaluate the effectiveness of Generative Pre-trained Transformer (GPT)-4 in facilitating human-robot communication.

Main Methods:

• Development of a Unity Virtual Reality (VR) simulation framework for a multi-robot testbed.
• Implementation of individual GPT cores for simulated robot agents, using OpenAI's function calling for natural language to action mapping.
• Conducting a user study with 12 participants to observe interaction strategies and communication effectiveness.

Main Results:

• Users often exhibit preconceived notions about conversing with robots, limiting exploration of advanced language capabilities.
• Participants who actively explored the LLM's capabilities experienced more natural and human-like communication.
• The framework successfully bridged natural language input with structured robot actions.

Conclusions:

• Future systems should encourage users to explore the full cognitive and language potential of LLM-powered robots.
• Lessons learned provide insights for designing and implementing effective human-robot teaming systems.
• Verbal communication, enhanced by LLMs, can significantly improve human-robot collaboration.