Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Making.

Area of Science:

• Artificial Intelligence
• Human-Computer Interaction
• Sociotechnical Systems

Background:

• Large Language Models (LLMs) are increasingly integrated into high-stakes decision-making processes.
• Understanding the sociotechnical factors governing human-LLM interaction is critical for effective collaboration.
• Existing research lacks a structured framework for analyzing human-LLM roles in decision-making.

Purpose of the Study:

• To introduce and define the concept of human-LLM archetypes as recurring sociotechnical interaction patterns.
• To identify and categorize diverse human-LLM archetypes from existing literature.
• To evaluate the impact of different archetypes on decision outcomes in clinical diagnostics.

Main Methods:

• Conducted a scoping literature review and thematic analysis of 113 papers on LLM-supported decision-making.
• Developed 17 distinct human-LLM archetypes based on identified interaction patterns.
• Evaluated archetype effects using real-world clinical diagnostic case studies.

Main Results:

• Identified 17 distinct human-LLM archetypes structuring collaborative decision-making.
• Demonstrated that the choice of human-LLM archetype significantly influences LLM outputs and diagnostic outcomes.
• Highlighted tradeoffs related to decision control, social hierarchies, and information requirements.

Conclusions:

• The selection of a human-LLM interaction archetype critically impacts AI-assisted decision-making.
• Designers of human-AI systems must consider these archetypes to manage risks and optimize performance.
• Further research is needed to explore the implications of specific archetypes in various high-stakes domains.