A quantitative analysis of accuracy, reliability and bias in judgements of functional analyses

Functional analysis can be considered a diagnostic assessment that behavior analysts use to determine behavioral function. Such a diagnosis ultimately requires a yes or no decision (i.e., a variable maintains a behavior, or it does not) that is determined by both subjective (clinical judgement) and objective (data) variables. Accurate and reliable identification of function is essential for successful treatment, yet behavior analysts' interpretation of data relies on their ability to detect visual differences in graphed data. Some research indicates that behavior analysts have questionable reliability in their visual analysis. To further examine the reliability, accuracy, and bias in visual analysis of functional analysis graphs, we simulated functional analysis results and surveyed 121 BCBA-Ds experienced in visual analysis. We then examined reliability of responses and used a signal detection theory approach to analyze accuracy and bias. Findings suggest that reliability and accuracy of judgements are questionable, and exploration of decision aids is warranted.