The mathematics under the hood: optimization methods, convergence guarantees, and the theory that explains when and why deep learning works. Explainers that take proofs seriously without losing the reader.
A fraud analyst at a bank trains a random forest of a hundred trees on card transactions. Some of those trees happen to be excellent at spotting one rare kind of fraud and mediocre at everything else. Others are the…
Fractional Gradients Reweight Random Forest Trees by Class Read More »
Splitting an image into meaningful regions can come down to a surprisingly old trick, picking a few brightness cutoffs and sorting every pixel into the band it falls in. It works well until you want more than a couple of…
MAAPO Optimizes Image Thresholds With Membrane Computing Read More »
Ask any expressivity theorist why deep networks are believed to handle high dimensional problems well, and Barron’s 1993 paper comes up almost immediately. Andrew Barron showed that a shallow network with N neurons can approximate a certain class of functions,…
How a Fake Fourier Basis Solves the Curse of Dimensionality in Neural Networks Read More »
Ordinary stochastic gradient descent assumes you can compute an honest, unbiased estimate of the gradient at every step. Pick a random example or a random minibatch, differentiate the loss, and step in the opposite direction. That assumption quietly breaks in…
How Biased Stochastic Gradients Still Generalize Well Read More »
Support vector machines have been a workhorse of classification since Cortes and Vapnik introduced them in 1995. Feed the model a set of labeled examples and it finds a hyperplane that separates the classes with the widest possible margin. That…
Why Support Vector Machines Finally Get Honest Confidence Intervals Read More »
A biologist staring at a flow cytometry readout from a single cell experiment wants to know which protein in a signaling cascade is actually driving which. Not correlated with. Driving. That distinction, cause versus association, is the entire reason causal…
Dimension Reduction Speeds Up Causal Graph Discovery Read More »
Stochastic Bregman proximal gradient methods drop the Lipschitz assumption behind SGD’s convergence proofs, match optimal sample complexity, and resist gradient explosion where standard optimizers collapse.
The idea that batch size shapes generalization dates back at least to Keskar et al. (2017), who showed empirically that large-batch training reliably finds sharper minima and those minima tend to generalize worse. The explanation became common wisdom: stochastic noise…
Why Batch Size Changes What Your Neural Network Learns Read More »
Expected Improvement has a lot going for it. It has a closed-form expression, it naturally trades off exploration and exploitation, and decades of practitioners have found that it just works. Jones, Schonlau, and Welch popularized it in 1998, and it…
When Expected Improvement Falls Short and What EIC Does About It Read More »
A research team from the Chinese University of Hong Kong and Florida State University has delivered the first unified convergence theory for a broad class of score based generative models in 2-Wasserstein distance, and it shows that the forward process…
Wasserstein Convergence Guarantees for Score-Based Generative Models Read More »