How Biased Stochastic Gradients Still Generalize Well
Ordinary stochastic gradient descent assumes you can compute an honest, unbiased estimate of the gradient at every step. Pick a random example or a random minibatch, differentiate the loss, and step in the opposite direction. That assumption quietly breaks in…
How Biased Stochastic Gradients Still Generalize Well Read More »










