Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Videos

AUBER: Automated BERT regularization.

Hyun Dong Lee1, Seongmin Lee2, U Kang2

  • 1Columbia University, New York, NY, United States of America.

Plos One
|June 28, 2021
PubMed
Summary
This summary is machine-generated.

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Synthesis and Characterization of Chemically Stable N7-dG Estrone and Catechol Adducts.

Molecules (Basel, Switzerland)·2026
Same author

Stiefel Manifold Dynamical Systems for Tracking Representational Drift.

bioRxiv : the preprint server for biology·2026
Same author

CatDive: A simple yet effective method for maximizing category diversity in sequential recommendation.

PloS one·2026
Same author

Mutagenic Potentials of DNA Interstrand Cross-Links Induced by 7,8-Dihydro-8-Oxoadenine.

Molecules (Basel, Switzerland)·2026
Same author

Fusion yield enhancement via secondary beam-target reactions in laser-cluster experiments.

Scientific reports·2026
Same author

Development of Metal-Enhanced Fluorescence Nanorods on Micro Post Arrays for Portable Detection of Human Semen Biomarkers.

Micromachines·2025
Same journal

Thymidylate synthase inhibitory drugs induce p53-dependent pathways differently.

PloS one·2026
Same journal

Top-down and bottom-up attention for joint pattern classification and reconstruction.

PloS one·2026
Same journal

Short- and long-term scaling behavior of blood pressure and pulse arrival time during sleep in healthy controls and patients with obstructive sleep apnea.

PloS one·2026
Same journal

Double DQN-based secrecy energy efficiency and fairness performance in IRS-assisted NOMA systems with friendly jamming.

PloS one·2026
Same journal

10 recommendations for strengthening citizen science for improved societal and ecological outcomes: A co-produced analysis of challenges and opportunities in the 21st century.

PloS one·2026
Same journal

Paying in public: Peer effects, impression management, and willingness to pay on digital payment platforms.

PloS one·2026
See all related articles

Automated BERT Regularization (AUBER) uses reinforcement learning to prune BERT attention heads, improving performance on small datasets. This method outperforms existing techniques by automatically identifying and removing suboptimal attention heads for better model generalization.

Area of Science:

  • Natural Language Processing (NLP)
  • Machine Learning
  • Deep Learning

Background:

  • BERT models excel in NLP but are prone to overfitting with limited training data.
  • Current regularization methods for BERT, like attention head pruning, often lack optimization and direct performance goals.

Purpose of the Study:

  • To introduce AUBER, an automated method for regularizing BERT models.
  • To enhance BERT's performance and generalization, especially on datasets with few training instances.

Main Methods:

  • Leveraging reinforcement learning to automatically identify and prune suboptimal attention heads in BERT.
  • Employing a low-dimensional state representation and a dually-greedy approach to reduce model complexity and search space during training.

Related Experiment Videos

Main Results:

  • AUBER achieved superior performance compared to existing pruning methods, with improvements up to 9.58%.
  • Ablation studies confirmed the effectiveness of the specific design choices within the AUBER framework.

Conclusions:

  • AUBER offers an effective and automated approach to regularize BERT, significantly boosting performance on data-scarce scenarios.
  • The proposed method addresses limitations of manual pruning by optimizing head selection through reinforcement learning.