Stimulus uncertainty and relative reward rates determine adaptive responding in perceptual decision-making

In dynamic environments, animals must select actions based on sensory input as well as expected positive and negative consequences. This type of behavior is typically studied using perceptual decision making (PDM) tasks. The arguably most influential framework for describing the cognitive processes underlying PDM is signal detection theory (SDT). One central assumption of SDT is that observers make perceptual decisions by comparing sensory evidence to a static decision criterion. However, mounting evidence suggests that the criterion is in fact highly dynamic and that observers adjust it flexibly according to task demands. Nevertheless, the mechanisms by which observers integrate stimulus and reward information for adaptive criterion learning remain not well understood. Here, we systematically investigated the factors influencing criterion setting at the single-trial level. To that end, we first specified three SDT-based models that learn either from reward, reward omission, or both. Next, by concomitantly manipulating stimulus and reward probabilities, we constructed experimental conditions in which these models make divergent predictions. Finally, we subjected rats and pigeons to a PDM task comprising these conditions. We find that subjects adopted decision criteria that maximize total reward in all experimental conditions. Detailed behavioral analyses reveal that criterion learning is driven by the integration of rewards, not reward omissions, and that reward integration is influenced by two additional factors: first, the degree of stimulus uncertainty, and second, the difference in the relative reward rates (rather than the absolute reward rates) between the choice alternatives. A model incorporating these factors accounts well for criterion dynamics across experimental conditions for both species and links signal detection theory to a learning mechanism operating at the level of single trials which, in the steady state, produces behavior similar to the matching law, a central tenet of learning theory.