The mathematics under the hood: optimization methods, convergence guarantees, and the theory that explains when and why deep learning works. Explainers that take proofs seriously without losing the reader.
A sentiment classifier predicts a positive review correctly, yet its attention concentrates on a generic word rather than the word carrying the sentiment. Another model makes the same decision and attends elsewhere. Accuracy alone leaves the difference unexplained. The optimizer…
Mirror Descent Changes Which Tokens Attention Selects Read More »
A researcher tests a complex neuron on a target that an ordinary ReLU neuron can learn. The complex model has more flexibility, yet its loss settles into a slow decline. The extra phase parameter can represent useful behavior. In this…
Complex Neurons Learn Faster With Phase Reparameterization Read More »
A reviewer reads a results table with four columns for a ten class image benchmark. Accuracy 0.912, micro precision 0.912, micro recall 0.912, micro F1 0.912. The authors present them as four pieces of evidence. The reviewer suspects they are…
Classification Metrics Unified by One Matrix Algebra Read More »
You are fitting a tensor decomposition. Every factor update solves a manageable problem, the reconstruction error keeps falling, and the run eventually seems to settle. Then another initialization takes much longer. Adding a small quadratic penalty helps, until you impose…
Block Riemannian Optimization and RBMM Guarantees Read More »
An analyst has five thousand unlabelled records and one colleague who actually understands them. The colleague will not label five thousand rows. They will, if asked politely, look at two records at a time and say whether they belong together.…
FALCON Asks Better Clustering Questions With MMD Read More »
A researcher increases the hidden width of a small classifier because the proof seems to demand it. Training already works with far fewer neurons. The gap raises an awkward question. Does the network need all those parameters, or does the…
Shallow ReLU Networks Get Sharper Generalization Bounds Read More »
A fraud analyst at a bank trains a random forest of a hundred trees on card transactions. Some of those trees happen to be excellent at spotting one rare kind of fraud and mediocre at everything else. Others are the…
Fractional Gradients Reweight Random Forest Trees by Class Read More »
Splitting an image into meaningful regions can come down to a surprisingly old trick, picking a few brightness cutoffs and sorting every pixel into the band it falls in. It works well until you want more than a couple of…
MAAPO Optimizes Image Thresholds With Membrane Computing Read More »
Ask any expressivity theorist why deep networks are believed to handle high dimensional problems well, and Barron’s 1993 paper comes up almost immediately. Andrew Barron showed that a shallow network with N neurons can approximate a certain class of functions,…
How a Fake Fourier Basis Solves the Curse of Dimensionality in Neural Networks Read More »
Ordinary stochastic gradient descent assumes you can compute an honest, unbiased estimate of the gradient at every step. Pick a random example or a random minibatch, differentiate the loss, and step in the opposite direction. That assumption quietly breaks in…
How Biased Stochastic Gradients Still Generalize Well Read More »