Out-of-context and out-of-scope: Manipulating large language models through minimal instruction set modifications.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• Emerging reasoning capabilities in large language models (LLMs) require alignment with human intentions.
• Out-of-context reasoning allows LLMs to adopt behaviors from descriptive information without examples.
• Assessing the benign nature of LLM behavioral patterns is challenging due to opaque training data.

Purpose of the Study:

• To investigate the embedding and triggering mechanisms of user-defined behaviors in LLMs.
• To understand the influence of training data and prompting strategies on LLM behavior.
• To evaluate the potential for manipulation of LLM response patterns.

Main Methods:

• Fine-tuning LLMs with minimal, hidden behavioral descriptions within a larger instruction set.
• Mathematical motivation and empirical validation of signal diffusion and inference in training data.
• Analysis of prompting strategies and token reinforcement for behavior triggering.

Main Results:

• LLMs can infer and adopt response patterns from diffused signals in noisy training data.
• Behavioral embedding is sensitive to minimal instruction set modifications.
• Triggering these embedded behaviors heavily depends on specific prompting strategies and token reinforcement.

Conclusions:

• LLMs exhibit susceptibility to manipulation through subtle modifications in training data and prompting.
• The emergence of specific behaviors in LLMs can be controlled and triggered under certain conditions.
• Unmonitored use of LLMs with inaccessible training data poses unforeseen risks due to potential behavioral manipulation.