Apples and oranges: Conceptual review as task analysis method

Conceptual review is a method to address issues of task comparability and task validity in cognitive neuroscience. Meta-analyses within cognitive neuroscience (CNS) as well as integration of neuroscientific findings with findings from adjacent disciplines both involve gathering studies that have purportedly investigated the same mental concept. After all, it is no use comparing apples and oranges. Tasks, and in particular the experimental contrasts implemented through tasks, determine whether studies are in fact comparable. Yet studies tend to be grouped together or kept apart based on the mental label researchers have applied and unfortunately, labels are an unreliable proxy for experimental contrasts. Different contrasts may receive the same label: 'working memory' studies rely on a variety of contrasts, derived from a variety of tasks. Vice versa, the same contrast may receive different labels: 'task switching' and 'working memory' studies can be exactly the same in terms of their experimental contrast. Label use thus obscures comparability problems. What is more, even when experimental contrasts are comparable, they may be invalid operationalizations of the mental label attached to them. In this paper, I introduce conceptual review as a method for task analysis. It can stand on its own or be combined with a cognitive ontology. Conceptual review applies philosophical strategies for analysing concepts to methodological choices in CNS studies, to uncover their conceptual implications. Conceptual review thus sheds light on the precise concept that was studied and thereby, on the comparability of CNS studies and the validity of tasks.