What's in a Prompt? Assignment Prompt Cues Influence Data Analysis, Critique, and Inclusion of Follow-up Experiments

An important goal of STEM education is to develop the ability to think like a scientist: to critically analyze evidence, form evidence-based claims, and think of follow-up experiments. When designing assignments to foster these skills, it is important to consider the effect of prompt design. To investigate this effect, we provided 110 upper-division biology majors with three prompts that asked them to interpret the same figure from a scientific paper but differed in the cues and in the context of the task, presenting it as an argument-building, Sense-making, or Grant-reviewer activity. Responses by biology faculty served as a benchmark of thinking like a scientist. The Argument and Sense-making prompts elicited more complete interpretations of the data than the Grant reviewer prompt, but students responding to all three prompts struggled to form claims based on multiple panels of the figure. Inclusion of cues to critique the data and offer follow-up experiments in the Sense-making and Grant reviewer prompts resulted in most responses including these scientific practices. The Grant reviewer prompt elicited critical evaluation of the data to support an argument for or against "funding." Our results indicate that prompt design substantially influences what scientific skills students will demonstrate.