Task-dependent color representation in convolutional neural networks

Convolutional neural networks (CNNs) trained for image categorization are known to have color-selective units that are tuned to particular colors. Analogously to the analysis of color representations in humans using brain imaging data, the representation of color within layers of a trained network can be characterized by constructing representational dissimilarity matrices (RDMs) and by using multidimensional scaling (MDS) to visualize geometric representational color spaces. Human color representations show flexibility dependent on the task. We trained CNNs on a set of simple chromatic stimuli but varied the "task" to require either color categorization, an analog of a color appearance rating, or luminance or spatial judgments that may not require color at all. We found that color representations within trained networks differed reliably and distinctively between task conditions, and that structured representations developed for color-relevant training conditions that were appropriate to the task. Color representations for different task conditions diverged through network layers toward the output layer, but they were significantly different even for layer 2 near the input layer. The variance between network instances was lowest for color-relevant tasks. For two of the tasks initially assumed to be "color-irrelevant," reliable and distinctive color representations developed. For these tasks requiring luminance or spatial judgments, color must provide a useful cue even though it is not required in the network output. These results or similar models may be used to generate hypotheses for how color representations in the human brain vary with task (whether tasks obviously require color or not), which could be tested using neuroimaging.