MacBehaviour: An R package for behavioural experimentation on large language models.

Area of Science:

• Cognitive Science
• Artificial Intelligence
• Computational Linguistics

Background:

• Large language models (LLMs) and LLM-powered chatbots are increasingly studied using psychological experiment methodologies.
• Existing research often treats LLMs as participants, necessitating efficient tools for behavioral experimentation.

Purpose of the Study:

• To introduce "MacBehaviour," an R package designed to streamline the process of conducting psychological experiments with LLMs.
• To provide researchers with a user-friendly tool for experiment design, stimuli presentation, model behavior manipulation, and data logging.

Main Methods:

• Developed the "MacBehaviour" R package, enabling interaction with over 100 LLMs (e.g., GPT, Claude, Gemini, Llama).
• Implemented functions for experiment setup, stimulus delivery, response logging, and token probability analysis.
• Validated the package through three experiments on GPT-3.5 Turbo, Llama-2-7b-chat-hf, and Vicuna-1.5-13b.

Main Results:

• The "MacBehaviour" package successfully facilitated LLM behavioral experimentation.
• Replication of the sound-gender association study demonstrated that tested LLMs exhibit human-like tendencies to infer gender from novel personal names based on phonology.
• LLMs showed consistent patterns mirroring human cognitive biases in gender association.

Conclusions:

• "MacBehaviour" is a valuable, user-friendly R package that simplifies and standardizes machine behavior studies.
• The package enhances accessibility for researchers investigating LLM cognition and behavior.
• Findings confirm LLMs' capacity for human-like phonological-based gender inference, advancing our understanding of AI cognition.