The TrollLabs open hackathon dataset: Generative AI and large language models for prototyping in engineering design

Area of Science:

• Engineering Design
• Artificial Intelligence
• Human-Computer Interaction

Background:

• The integration of Generative AI into engineering design processes is rapidly evolving.
• Comparative studies assessing AI performance against human experts in design tasks are crucial.
• The TrollLabs Open dataset provides a unique resource for such analyses.

Purpose of the Study:

• To compare the design practices and outcomes of a Generative AI (ChatGPT) team against experienced human engineering students.
• To curate and release a comprehensive dataset from a prototyping hackathon for open research.
• To provide a benchmark for AI capabilities in early-stage engineering design.

Main Methods:

• A prototyping hackathon was conducted where one team was autonomously directed by ChatGPT, and five baseline teams comprised experienced engineering students.
• Participants designed and prototyped a device to launch a NERF dart.
• Data collection included prototype attributes (116 entries, 433 connections), a transcript of AI-human interaction (97 exchanges), and objective team performance metrics.

Main Results:

• The AI-directed team achieved 2nd place out of six competing teams.
• The dataset enables comparative analysis of design strategies and outcomes between AI and human teams.
• Prototypes and interaction logs were captured using the Pro2booth platform.

Conclusions:

• The TrollLabs Open dataset is the first to directly compare Generative AI and human teams in an engineering design context.
• The findings suggest Generative AI can perform competitively in design challenges.
• This resource is valuable for researchers, students, educators, and professionals exploring AI integration in design.