Related Experiment Video
Updated: Aug 2, 2026
12:03
Myocardial Infarction and Functional Outcome Assessment in Pigs
Published on: April 25, 2014
28.7K
The fallacy of inscrutability
1School of Information, University of California Berkeley, Berkeley, CA, USA kroll@berkeley.edu.
Summary
Algorithms are understandable technologies, not inscrutable black boxes. Policy should focus on formalizing system assumptions and design choices for trustworthy AI governance, rather than demanding explanations for every behavior.
Area of Science:
- Computer Science
- Ethics
- Public Policy
Background:
- Software systems face criticism for being inscrutable
- black boxes
- obscuring decision-making logic.
- Power dynamics can exacerbate this opacity.
- However, algorithms are fundamentally understandable technologies.
Purpose of the Study:
- To investigate the nature of inscrutability and opacity in software systems.
- To analyze the role of power dynamics in creating opaque systems.
- To evaluate proposed remedies from computer ethics and privacy by design.
Main Methods:
- Conceptual analysis of software systems as understandable technologies.
- Examination of how power structures contribute to system opacity.
- Review of interdisciplinary approaches to address algorithmic inscrutability.
Main Results:
- Algorithms are inherently understandable, designed with specific purposes and assumptions.
- Inscrutability often arises from power dynamics, not inherent technological complexity.
- No system is wholly inscrutable, as creators and operators determine fitness for use.
Conclusions:
- Policy should reject the notion of necessarily inscrutable systems.
- Effective AI governance requires rigorous science and engineering for verifiable trustworthiness.
- Formalizing and disclosing system assumptions, choices, and adequacy determinations is crucial.
Related Concept Videos
02:40The Scientific Method
Research is what makes the difference between facts and opinions. Facts are observable realities, and opinions are personal judgments, conclusions, or attitudes that may or may not be accurate. In the scientific community, facts can be established only using evidence collected through empirical research.
01:37Reason and Intuition
The human brain processes information for decision-making using one of two routes: an intuitive system and a rational system (Epstein, 1994; popularized by Kahneman, 2011 as System 1 and System 2, respectively). The intuitive system is quick, impulsive, and operates with minimal effort, relying on emotions or habits to provide cues for what to do next, while the rational system is logical, analytical, deliberate, and methodical. Research in neuropsychology suggests that the brain can only use...
01:12Hindsight Biases
Hindsight bias leads you to believe that the event you just experienced was predictable, even though it really wasn’t. In other words, you knew all along that things would turn out the way they did. Can you relate this to the phrase "Hindsight is 20/20" now?
01:14Fundamental Attribution Error
According to some social psychologists, people tend to overemphasize internal factors as explanations—or attributions—for the behavior of other people. They tend to assume that the behavior of another person is a trait of that person, and to underestimate the power of the situation on the behavior of others. They tend to fail to recognize when the behavior of another is due to situational variables, and thus to the person’s state. This erroneous assumption is called the fundamental attribution...
03:50The Scientific Method
Chemistry is an empirical science. Scientists often pose questions to understand the chemistry in everyday life and seek answers to these questions. To achieve this, scientists follow a definitive series of steps that together make up the Scientific Method. This approach involves making observations, asking questions, building a hypothesis, conducting experiments, analyzing results, and forming a conclusion.
01:29Introspection
Introspection, long upheld as a reliable route to self-knowledge, involves examining one's thoughts, emotions, and mental processes. It underpins many psychological practices, from mindfulness meditation to psychotherapy and self-help strategies. However, empirical evidence challenges the accuracy of introspection as a means of understanding oneself.Limitations of Introspective InsightSeminal work by Nisbett and Wilson demonstrated that individuals are frequently unaware of the true causes...