Related Experiment Video
Updated: May 9, 2025
13:00
Measuring Attention and Visual Processing Speed by Model-based Analysis of Temporal-order Judgments
Published on: January 23, 2017
9.8K
State-of-the-Art: The Temporal Order of Benchmarking Culture
1Department of Geography, Durham University, Lower Mountjoy, South Road, Durham, DH1 3LE UK.
Digital Society : Ethics, Socio-Legal and Governance of Digital Technology
|May 5, 2025
Summary
Machine learning benchmarking creates a present-focused research culture by normalizing progress and emphasizing the current state-of-the-art. This temporal structure influences research direction and expectations.
Area of Science:
- Computer Science
- Philosophy of Science
Background:
- Machine learning research heavily relies on benchmarking and evaluation.
- Existing critiques often focus on statistical validity and representativeness of benchmarks.
Purpose of the Study:
- To analyze the temporal values and expectations produced by machine learning's benchmarking culture.
- To investigate how benchmarking influences the rhythm and direction of research.
Main Methods:
- Commentary and theoretical analysis.
- Formulation of two hypotheses: normalizing research and extrapolation.
- Examination of epistemic values within temporal structures.
Main Results:
- Benchmarking normalizes research by providing disciplining and motivating functions, minimizing conflict.
- The incremental progress in benchmarking is oriented towards the present state-of-the-art (SOTA), not the future.
- This leads to a presentist temporality in machine learning evaluation.
Conclusions:
- Machine learning's benchmarking culture fosters a presentist orientation.
- This temporal structure shapes research practices and expectations.
- Further examination of temporal values in AI evaluation is warranted.
Related Concept Videos
03:26Framing Effects
7.3K
Information is everywhere and its presentation—such as how and when items are presented—can impact our perceptions and decisions surrounding the info. This broad concept umbrellas framing effects—influences that occur due to the way information is framed in its appearance, whether it’s purely the order or the specific wording of a message. Let’s take a look at numerous ways in which two versions of something can objectively say the same thing, yet we respond in...
7.3K
00:52Social Proof
27.2K
Social proof is a form of persuasion based on comparison and conformity. People compare their behavior and actions to what others are doing and will change to conform to do what their peers do.
27.2K
01:26Cultural Influences on Personality
81
Individualist and collectivist cultures emphasize different core values, shaping personality in distinct ways. In individualist cultures, such as those in the United States, England, and Australia, people prioritize independence, competition, and personal achievement. These societies tend to promote self-focused traits, with individuals often reporting higher levels of self-esteem. In contrast, collectivist cultures, commonly found in regions like Asia, Africa, and South America, emphasize...
81
02:16Stereotype Content Model
13.9K
The Stereotype Content Model (SCM) was first proposed by Susan Fiske and her colleagues (Fiske, Cuddy, Glick & Xu, 2002; see also Fiske, 2012 and Fiske, 2017). The SCM specifies that when someone encounters a new group, they will stereotype them based on two metrics: warmth—or that group’s perceived intent, and how likely they are to provide help or inflict harm—and competence—or their ability to carry out that objective. Depending on the warmth-competence...
13.9K
02:05Self-Presentation: Self-Monitoring and Self-Handicapping
38.6K
People can go to great lengths to protect their self-image and present themselves in ways that they want others to see them. Sociologist Erving Goffman presented the idea that a person is like an actor on a stage. Calling his theory dramaturgy, Goffman believed that we use “impression management” to present ourselves to others as we hope to be perceived. Each situation is a new scene, and individuals perform different roles depending on who is present (Goffman, 1959). Think about...
38.6K
04:49Robbers Cave
14.2K
During the 1950s, the landmark Robbers Cave experiment demonstrated that when groups must compete with one another, intergroup conflict, hostility, and even violence may result. At the Oklahoman summer camp, two troops of boys—termed the Rattlers and the Eagles—took part in a week-long tournament. During this time, their negativity culminated in derogatory name-calling, fistfights, and even vandalism and destruction of property. However, this work also revealed that such tension...
14.2K