Related Experiment Video
Updated: Sep 22, 2025
08:58
Artificial Intelligence Approaches to Assessing Primary Cilia
Published on: May 1, 2021
3.7K
Reliance on metrics is a fundamental challenge for AI
Rachel L Thomas1, David Uminsky2
1Queensland University of Technology, Brisbane, QLD, Australia.
Patterns (New York, N.Y.)
|May 24, 2022
Summary
Unthinking pursuit of metric optimization causes real-world harms. An evidence-based framework using multiple metrics, audits, qualitative data, and stakeholder input can mitigate these risks.
Area of Science:
- Computer Science
- Social Science
- Ethics
Background:
- Metric optimization in AI systems can lead to unintended negative consequences.
- Proxies for user satisfaction, like watch time, often fail to capture true user experience.
Purpose of the Study:
- To review harms caused by unthinking metric optimization.
- To propose a framework for mitigating these harms.
Main Methods:
- Case study analysis of real-world harms.
- Development of an evidence-based mitigation framework.
Main Results:
- Identified harms include algorithmic radicalization, unfair teacher dismissals, and flawed essay grading.
- Proposed framework includes using multiple metrics, external audits, qualitative data, and stakeholder involvement.
Conclusions:
- A multi-faceted approach is necessary to ensure responsible AI development and deployment.
- Mitigating harms requires moving beyond single-metric optimization to a more holistic evaluation.
Related Concept Videos
01:29Measures of Intelligence
7.9K
Psychologists measure intelligence by using standardized tests that produce a score known as the intelligence quotient or IQ. To understand IQ tests, it's important to recognize the key principles behind their construction: validity, reliability, and standardization.
Validity refers to how well a test measures what it claims to measure. An intelligence test should accurately assess intelligence rather than another characteristic, like anxiety. Criterion validity is one way to evaluate this;...
Validity refers to how well a test measures what it claims to measure. An intelligence test should accurately assess intelligence rather than another characteristic, like anxiety. Criterion validity is one way to evaluate this;...
7.9K
01:11Statistical Analysis: Overview
7.8K
When we take repeated measurements on the same or replicated samples, we will observe inconsistencies in the magnitude. These inconsistencies are called errors. To categorize and characterize these results and their errors, the researcher can use statistical analysis to determine the quality of the measurements and/or suitability of the methods.
One of the most commonly used statistical quantifiers is the mean, which is the ratio between the sum of the numerical values of all results and the...
One of the most commonly used statistical quantifiers is the mean, which is the ratio between the sum of the numerical values of all results and the...
7.8K
01:23Binet's Contribution to Measures of Intelligence
1.4K
Alfred Binet, along with his student Théophile Simon, was tasked by the French Ministry of Education in 1904 to create a method for identifying students who struggled to learn through conventional classroom instruction. This initiative aimed to address overcrowding by placing such students in specialized schools. Binet and Simon developed an intelligence test comprising 30 tasks, ranging from simple commands, like touching one's nose or ear, to more complex tasks, such as drawing...
1.4K
01:28Accuracy, limits, and approximation
560
Accuracy, limits, and approximations are common in many fields, especially in engineering calculations. These concepts are imperative for ensuring that a given value is as close as possible to its true value.
Accuracy is defined as the closeness of the measured value to the true or actual value. In engineering mechanics, repeated measurements are taken during theoretical or experimental analyses to ensure that the result is precise and accurate.
The accuracy of any solution is based on the...
Accuracy is defined as the closeness of the measured value to the true or actual value. In engineering mechanics, repeated measurements are taken during theoretical or experimental analyses to ensure that the result is precise and accurate.
The accuracy of any solution is based on the...
560
02:16Stereotype Content Model
14.9K
The Stereotype Content Model (SCM) was first proposed by Susan Fiske and her colleagues (Fiske, Cuddy, Glick & Xu, 2002; see also Fiske, 2012 and Fiske, 2017). The SCM specifies that when someone encounters a new group, they will stereotype them based on two metrics: warmth—or that group’s perceived intent, and how likely they are to provide help or inflict harm—and competence—or their ability to carry out that objective. Depending on the warmth-competence...
14.9K
01:25Distribution Reliability and Automation
167
Distribution reliability in electrical power systems is critical for ensuring an uninterrupted power supply to consumers at minimal cost. According to IEEE Standard Terms, reliability is the probability that a device will function without failure over a specified time period or amount of usage. For electric power distribution, this translates to maintaining continuous power supply and addressing customer concerns over power outages. Several indices, as defined by IEEE Standard 1366-2012, are...
167