Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Limits to Natural Selection01:38

Limits to Natural Selection

35.4K
Organisms that are well-adapted to their environment are more likely to survive and reproduce. However, natural selection does not lead to perfectly adapted organisms. Several factors constrain natural selection.
35.4K
The Evidence for Evolution02:55

The Evidence for Evolution

48.5K
Genetic variations accumulating within populations over generations give rise to biological evolution. Evolutionary changes can result in the formation of novel varieties and entire new species. These changes are responsible for the diverse forms of life inhabiting the planet. The evidence for evolution suggests that all living organisms descended from common ancestors.
48.5K
High-Performance Liquid Chromatography: Elution Process01:05

High-Performance Liquid Chromatography: Elution Process

1.6K
In High-Performance Liquid Chromatography (HPLC), the elution process is critical to the separation of analytes and the quality of chromatographic results. Elution describes how compounds move through the column and separate based on their interactions with the mobile and stationary phases. This process determines the resolution, peak shape, and retention times in the chromatogram, which are essential for identifying and quantifying components in complex mixtures. Understanding the elution...
1.6K
Nature and Nurture01:10

Nature and Nurture

22.5K
Many human characteristics, like height, are shaped by both nature—in other words, by our genes—and by nurture, or our environment. For example, chronic stress during childhood inhibits the production of growth hormones and consequently reduces bone growth and height. Scientists estimate that 70-90% of variation in height is due to genetic differences among individuals, and 10-30% of variation in height is due to differences in the environments that individuals experience,...
22.5K
Language01:16

Language

924
Language is a unique communication system that uses words and systematic rules to organize and transmit information. Unlike other forms of communication, which may involve postures, movements, odors, or vocalizations, language relies on symbols and grammar. This makes human communication distinct from that of other species, who also communicate but do not use language in the same way humans do.
Corballis and Suddendorf (2007) and Tomasello and Rakoczy (2003) highlight the role of language in...
924
Genome Annotation and Assembly03:36

Genome Annotation and Assembly

21.1K
The genome refers to all of the genetic material in an organism. It can range from a few million base pairs in microbial cells to several billion base pairs in many eukaryotic organisms. Genome assembly refers to the process of taking the DNA sequencing data and putting it all back together in a correct order to create a close representation of the original genome. This is followed by the identification of functional elements on the newly assembled genome, a process called genome annotation.
21.1K

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Ethics in digital phenotyping: considerations regarding Alzheimer's disease, speech and artificial intelligence.

Journal of medical ethics·2025
Same author

High-quality gene/disease embedding in a multi-relational heterogeneous graph after a joint matrix/tensor decomposition.

Journal of biomedical informatics·2022
Same author

Editor's introduction to the special section on the 7th Biomedical Linked Annotation Hackathon (BLAH7).

Genomics & informatics·2021
Same author

Bridging heterogeneous mutation data to enhance disease gene discovery.

Briefings in bioinformatics·2021
Same author

Natural Language Processing for Rapid Response to Emergent Diseases: Case Study of Calcium Channel Blockers and Hypertension in the COVID-19 Pandemic.

Journal of medical Internet research·2020
Same author

Editor's introduction to the special issue of the 6th Biomedical Linked Annotation Hackathon (BLAH6).

Genomics & informatics·2020

Related Experiment Video

Updated: Feb 16, 2026

Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
05:47

Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems

Published on: June 13, 2025

1.6K

Inter-Annotator Agreement and the Upper Limit on Machine Performance: Evidence from Biomedical Natural Language

Mayla Boguslav1, Kevin Bretonnel Cohen1

  • 1Computational Bioscience Program, University Colorado School of Medicine, Aurora, CO, USA.

Studies in Health Technology and Informatics
|January 4, 2018
PubMed
Summary

Inter-annotator agreement, a common measure for human-annotated data quality in natural language processing (NLP), is often assumed to be the performance ceiling for NLP systems. However, this study presents evidence challenging this long-held assumption, suggesting it may not be a strict upper bound.

Keywords:
Evaluation StudiesNatural Language ProcessingSupervised Machine Learning

More Related Videos

Using Eye Movements Recorded in the Visual World Paradigm to Explore the Online Processing of Spoken Language
09:27

Using Eye Movements Recorded in the Visual World Paradigm to Explore the Online Processing of Spoken Language

Published on: October 13, 2018

10.8K
Asthma Detection Research Based on Voice Signal Processing and Machine Learning
04:04

Asthma Detection Research Based on Voice Signal Processing and Machine Learning

Published on: July 22, 2025

1.1K

Related Experiment Videos

Last Updated: Feb 16, 2026

Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
05:47

Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems

Published on: June 13, 2025

1.6K
Using Eye Movements Recorded in the Visual World Paradigm to Explore the Online Processing of Spoken Language
09:27

Using Eye Movements Recorded in the Visual World Paradigm to Explore the Online Processing of Spoken Language

Published on: October 13, 2018

10.8K
Asthma Detection Research Based on Voice Signal Processing and Machine Learning
04:04

Asthma Detection Research Based on Voice Signal Processing and Machine Learning

Published on: July 22, 2025

1.1K

Area of Science:

  • Natural Language Processing
  • Computational Linguistics
  • Data Science

Background:

  • Human-annotated data is crucial for developing and evaluating natural language processing (NLP) systems.
  • Inter-annotator agreement (IAA) is the standard metric for assessing data quality.
  • A prevailing assumption posits that IAA represents the maximum achievable performance for NLP systems.

Purpose of the Study:

  • To investigate the foundational assumptions behind measuring inter-annotator agreement in NLP.
  • To examine the widespread belief that IAA dictates the upper limit of NLP system performance.
  • To empirically test whether IAA is indeed a strict performance ceiling.

Main Methods:

  • Historical analysis of the motivations for IAA measurement.
  • Empirical data collection and analysis of NLP task performance.
  • Comparative analysis of IAA metrics against system performance benchmarks.

Main Results:

  • The study traces the philosophical underpinnings of IAA measurement.
  • Data indicates that inter-annotator agreement does not consistently serve as an upper bound for NLP system performance.
  • Observed NLP system performance can exceed levels of human agreement.

Conclusions:

  • The assumption that inter-annotator agreement limits NLP system performance is not universally supported by empirical evidence.
  • Rethinking the role of IAA in NLP evaluation is necessary.
  • Future research should explore alternative or supplementary metrics for data quality and system performance assessment.