Building factorial prompt matrices using query targets and configurable factor classes: A methods framework for...

Area of Science:

• Computational Linguistics
• Human-Computer Interaction
• Social Sciences Research Methods

Background:

• Conversational AI (CAI) products are increasingly prevalent, yet methods for systematically studying their responses are unstandardized.
• Research into CAI behavior often lacks structured approaches for prompt variation.
• The need for controlled experimental designs to analyze CAI response patterns is growing.

Purpose of the Study:

• To formalize and introduce a methods framework for creating Factorial Prompt Matrices (FPMs).
• To provide a structured approach for social and behavioral scientists to conduct controlled research on CAI response behavior.
• To demonstrate the application of FPMs in analyzing CAI information retrieval, using medical prompts as a case study.

Main Methods:

• Developed a principled sampling spine of CAI query "target" items.
• Defined four classes of configurable factors: Large Language Model (LLM), product-system, user context, and query design.
• Crossed target items with factors to generate a factorial grid of distinct prompts for CAI interaction via APIs.

Main Results:

• Demonstrated the utility of FPMs in constructing and deploying systematic prompts to CAIs.
• Successfully applied FPMs to analyze CAI information retrieval behaviors, including web search invocation, using the ICD-11 as a prompt spine.
• Showcased FPMs' adaptability for both single-factor and multi-factor observational and experimental research designs.

Conclusions:

• Factorial Prompt Matrices offer a formalized and standardized data structure for prompt engineering in CAI research.
• The modular, four-class schema provides a flexible framework for organizing prompting protocols.
• FPMs facilitate rigorous, controlled investigations into CAI response behaviors, advancing scientific understanding.