Similarity in feature space dictates the efficiency of attentional selection during ensemble processing

Humans can rapidly and accurately extract statistical information about features of the visual environment, an ability referred to as ensemble perception. However, little is known about how ensemble estimates are affected when task-irrelevant and distracting feature information is present. Here, we tested how effectively feature-based attention-when tuned to a specific color-can select a single item set out of two intermixed ensembles of colored lines. Participants were instructed to report the average orientation of a target-colored item set, while ignoring a second differently colored set. To assess how representational overlap between the two sets impacts color-based selection, we systematically varied the orientation similarity between the relevant and irrelevant items. Our results showed that participants' orientation reports were reliably biased towards the irrelevant items, but interestingly, these biases were only observed when the item sets overlapped in orientation space. In a second experiment, using a visual mask to disrupt access to color information at different time points, we found that these biases were stronger when less time was available to process the stimuli. Together, these results suggest that ensemble representations are rapidly formed based on all available information in the relevant feature dimension, regardless of task relevance, and that selective attention weights and separates these ensemble representations at a relatively later processing stage. This selection appears highly effective when the underlying population activity generated by the two sets is separable along the to-be-estimated feature dimension, but is dampened when relevant and irrelevant ensemble representations overlap in feature space.