Search research articles

ABOUT JoVE

Overview Leadership Blog JoVE Help Center

AUTHORS

Publishing Process Editorial Board Scope & Policies Peer Review FAQ Submit

LIBRARIANS

Testimonials Subscriptions Access Resources Library Advisory Board FAQ

RESEARCH

JoVE Journal Methods Collections JoVE Encyclopedia of Experiments Archive

EDUCATION

JoVE Core JoVE Business JoVE Science Education JoVE Lab Manual Faculty Resource Center Faculty Site

Terms & Conditions of Use

Search research articles

Related Experiment Video

Updated: Oct 28, 2025

Gain-compensation Methodology for a Sinusoidal Scan of a Galvanometer Mirror in Proportional-Integral-Differential Control Using Pre-emphasis Techniques

Gain-compensation Methodology for a Sinusoidal Scan of a Galvanometer Mirror in Proportional-Integral-Differential Control Using Pre-emphasis Techniques

Published on: April 4, 2017

Stochastic Mirror Descent on Overparameterized Nonlinear Models.

Navid Azizan, Sahin Lale, Babak Hassibi

IEEE Transactions on Neural Networks and Learning Systems

|July 16, 2021

Summary

This summary is machine-generated.

Related Concept Videos

Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving

Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving

Mechanistic models play a crucial role in algorithms for numerical problem-solving, particularly in nonlinear mixed effects modeling (NMEM). These models aim to minimize specific objective functions by evaluating various parameter estimates, leading to the development of systematic algorithms. In some cases, linearization techniques approximate the model using linear equations.
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...

Propagation of Uncertainty from Random Error

Propagation of Uncertainty from Random Error

An experiment often consists of more than a single step. In this case, measurements at each step give rise to uncertainty. Because the measurements occur in successive steps, the uncertainty in one step necessarily contributes to that in the subsequent step. As we perform statistical analysis on these types of experiments, we must learn to account for the propagation of uncertainty from one step to the next. The propagation of uncertainty depends on the type of arithmetic operation performed on...

Model Approaches for Pharmacokinetic Data: Distributed Parameter Models

Model Approaches for Pharmacokinetic Data: Distributed Parameter Models

Pharmacokinetic models are mathematical constructs that represent and predict the time course of drug concentrations in the body, providing meaningful pharmacokinetic parameters. These models are categorized into compartment, physiological, and distributed parameter models.
The distributed parameter models are specifically designed to account for variations and differences in some drug classes. This model is particularly useful for assessing regional concentrations of anticancer or...

One-Compartment Open Model: Wagner-Nelson and Loo Riegelman Method for ka Estimation

One-Compartment Open Model: Wagner-Nelson and Loo Riegelman Method for k_a Estimation

This lesson introduces two critical methods in pharmacokinetics, the Wagner-Nelson and Loo-Riegelman methods, used for estimating the absorption rate constant (ka) for drugs administered via non-intravenous routes. The Wagner-Nelson method relates ka to the plasma concentration derived from the slope of a semilog percent unabsorbed time plot. However, it is limited to drugs with one-compartment kinetics and can be impacted by factors like gastrointestinal motility or enzymatic degradation.
On...

Propagation of Uncertainty from Systematic Error

Propagation of Uncertainty from Systematic Error

The atomic mass of an element varies due to the relative ratio of its isotopes. A sample's relative proportion of oxygen isotopes influences its average atomic mass. For instance, if we were to measure the atomic mass of oxygen from a sample, the mass would be a weighted average of the isotopic masses of oxygen in that sample. Since a single sample is not likely to perfectly reflect the true atomic mass of oxygen for all the molecules of oxygen on Earth, the mass we obtain from this...

Parameters Affecting Nonlinear Elimination: Zero-Order Input, First-Order Absorption and Two-Compartment Model

Parameters Affecting Nonlinear Elimination: Zero-Order Input, First-Order Absorption and Two-Compartment Model

Drugs administered through various routes can lead to nonlinear elimination, resulting in complex pharmacokinetic behaviors crucial to understanding efficacious drug dosing.
When a drug is administered through a constant intravenous infusion and eliminated via nonlinear pharmacokinetics, it follows zero-order input. For example, oral drugs undergo first-order absorption upon administration and are eliminated through nonlinear pharmacokinetics.
In the case of subcutaneously administered drugs,...

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by

Same author

HardFlow: Hard-Constrained Sampling for Flow-Matching Models Via Trajectory Optimization.

IEEE transactions on pattern analysis and machine intelligence·2026

Same author

Uncertainty Quantification in Detection Transformers: Object-Level Calibration and Image-Level Reliability.

IEEE transactions on pattern analysis and machine intelligence·2026

Same author

How to Query an Oracle? Efficient Strategies to Label Data.

IEEE transactions on pattern analysis and machine intelligence·2021

Same author

Low-Rank Riemannian Optimization for Graph-Based Clustering Applications.

IEEE transactions on pattern analysis and machine intelligence·2021

Same author

A Silicon Photonics Computational Lensless Active-Flat-Optics Imaging System.

Scientific reports·2020

Same author

Multiplexed identification, quantification and genotyping of infectious agents using a semiconductor biochip.

Nature biotechnology·2018

Same journal

Granular Ball-Based Noise-Resistant Fuzzy Multineighborhood Feature Selection via Label Enhancement and Feature Graph.

IEEE transactions on neural networks and learning systems·2026

Same journal

Fighting Evolving Spam With ARTMAP Models: A Noise-Resilient Online Detection Framework.

IEEE transactions on neural networks and learning systems·2026

Same journal

HyperSAT: Unsupervised Hypergraph Neural Networks for Weighted MaxSAT Problems.

IEEE transactions on neural networks and learning systems·2026

Same journal

Negation of Basic Belief Assignment in Multisource Information Fusion on Dempster-Shafer Theory With Applications in Pattern Classification.

IEEE transactions on neural networks and learning systems·2026

Same journal

Intervention Feasible Region and Driver Risk Capacity Aware Human-Machine Collaborative Safe Trajectory Planning.

IEEE transactions on neural networks and learning systems·2026

Same journal

A Unified Differential Denoising Learning Framework With a Pre-Trained Model and Fuzzy Graph Networks for Drug-Drug Interaction Prediction.

IEEE transactions on neural networks and learning systems·2026

See all related articles

Stochastic Mirror Descent (SMD) algorithms, including Stochastic Gradient Descent (SGD), converge to specific solutions in overparameterized models. Experiments show l-infinity regularization yields better generalization than SGD or l-one regularization.

Area of Science:

Machine Learning
Optimization Algorithms
Deep Learning Theory

Background:

Modern machine learning models are often overparameterized, leading to numerous solutions that perfectly fit training data.
Understanding which solutions are found by optimization algorithms and their impact on generalization is crucial.
Stochastic Mirror Descent (SMD) is a family of algorithms, including Stochastic Gradient Descent (SGD), used for optimization.

Purpose of the Study:

To investigate the convergence properties of Stochastic Mirror Descent (SMD) algorithms in overparameterized learning problems.
To analyze how initialization and learning algorithms influence the selection of interpolating solutions.
To compare the generalization performance of different SMD variants with varying regularization strategies.

Main Methods:

More Related Videos

Deep Neural Networks for Image-Based Dietary Assessment

Deep Neural Networks for Image-Based Dietary Assessment

Published on: March 13, 2021

Related Experiment Videos

Last Updated: Oct 28, 2025

Gain-compensation Methodology for a Sinusoidal Scan of a Galvanometer Mirror in Proportional-Integral-Differential Control Using Pre-emphasis Techniques

Gain-compensation Methodology for a Sinusoidal Scan of a Galvanometer Mirror in Proportional-Integral-Differential Control Using Pre-emphasis Techniques

Published on: April 4, 2017

Deep Neural Networks for Image-Based Dietary Assessment

Deep Neural Networks for Image-Based Dietary Assessment

Published on: March 13, 2021

Theoretical analysis of SMD convergence for overparameterized nonlinear models with fixed step sizes.
Empirical validation using experiments on MNIST and CIFAR-10 datasets.
Comparison of generalization performance using l1, l2 (SGD), and l-infinity regularization.

Main Results:

SMD algorithms approximately achieve implicit regularization by converging to solutions close to the minimum-potential interpolating solution.
Empirical results on MNIST and CIFAR-10 confirm theoretical predictions of SMD convergence behavior.
Experiments reveal that l-infinity regularization consistently outperforms SGD (l2) and l1 regularization in terms of generalization on CIFAR-10.

Conclusions:

SMD algorithms exhibit implicit regularization in sufficiently overparameterized nonlinear models.
The choice of regularization within SMD significantly impacts generalization performance, with l-infinity showing superior results.
Findings provide insights into the behavior of optimization algorithms in deep learning and guide the selection of effective regularization techniques.