Optimization & Learning Theory

The mathematics under the hood: optimization methods, convergence guarantees, and the theory that explains when and why deep learning works. Explainers that take proofs seriously without losing the reader.

How a Fake Fourier Basis Solves the Curse of Dimensionality in Neural Networks

How a Fake Fourier Basis Solves the Curse of Dimensionality in Neural Networks

Leave a Comment / Optimization & Learning Theory /

Analysis by the aitrendblend editorial team · Neural Network Approximation Theory ReLU networks Riesz basis Sobolev spaces Barron classes curse of dimensionality These two zig zag functions, one built to imitate cosine and one to imitate sine, turn out to behave like a Fourier basis and can be built exactly out of ReLU units. Every […]

How a Fake Fourier Basis Solves the Curse of Dimensionality in Neural Networks Read More »

How Biased Stochastic Gradients Still Generalize Well.

How Biased Stochastic Gradients Still Generalize Well

Leave a Comment / Optimization & Learning Theory /

Analysis by the aitrendblend editorial team · Optimization Theory and Mathematical Foundations · 14 min read stochastic gradient descent algorithmic stability generalization bounds Zeroth-order SGD Clipped-SGD excess risk Two families of biased gradient methods, one built from function values only and one built from clipped gradients, now share a single stability proof. A graduate student

How Biased Stochastic Gradients Still Generalize Well Read More »

Why Support Vector Machines Finally Get Honest Confidence Intervals

Why Support Vector Machines Finally Get Honest Confidence Intervals

Leave a Comment / Optimization & Learning Theory /

Analysis by the aitrendblend editorial team · Classical statistical learning theory support vector machine convolution smoothing Bahadur representation high dimensional inference hinge loss The sharp kink in the SVM hinge loss is the reason the classifier has never come with honest confidence intervals. Smoothing it changes that. A statistician training a support vector machine on

Why Support Vector Machines Finally Get Honest Confidence Intervals Read More »

Dimension Reduction Speeds Up Causal Graph Discovery.

Dimension Reduction Speeds Up Causal Graph Discovery

Leave a Comment / Optimization & Learning Theory, Machine Learning /

Pillar 5, graph learning and probabilistic methods. Analysis by the aitrendblend editorial team. Reading time about 14 minutes. Causal Discovery Sufficient Dimension Reduction PC Algorithm Directed Acyclic Graphs Kernel Methods RKHS A directed acyclic graph, the kind of structure this method tries to recover from observational data alone. A biologist staring at a flow cytometry

Dimension Reduction Speeds Up Causal Graph Discovery Read More »

Why Batch Size Changes What Your Neural Network Learns

Why Batch Size Changes What Your Neural Network Learns

1 Comment / Machine Learning, Math Applications, Optimization & Learning Theory /

Analysis by the aitrendblend editorial team January 2025 Machine Learning Research Optimization Feature Learning GD (left) settles near a dense interior minimum; SGD with b=1 (right) escapes to a single datapoint on the boundary. From Ghosh et al., JMLR 2025. Pick any mainstream guide to training neural networks and you will read the same advice:

Why Batch Size Changes What Your Neural Network Learns Read More »

When Expected Improvement Falls Short and What EIC Does About It.

When Expected Improvement Falls Short and What EIC Does About It

Leave a Comment / Machine Learning, Math Applications, Optimization & Learning Theory /

Practical AI Bayesian Optimization Analysis by the aitrendblend editorial team Published in JMLR 26 (2025) Cumulative regret curves from the EIC paper (Hu et al., JMLR 2025). EIC keeps pace with GP-UCB while closing the gap on traditional EI. Every machine learning practitioner who has tuned a neural network with Bayesian optimization has silently trusted

When Expected Improvement Falls Short and What EIC Does About It Read More »

Wasserstein Convergence Guarantees for Score-Based Generative Models.

Wasserstein Convergence Guarantees for Score-Based Generative Models

Leave a Comment / Machine Learning, Generative & Diffusion Models, Optimization & Learning Theory /

Generative Models · Journal of Machine Learning Research 26 (2025) 1 to 54 · 16 min read A research team from the Chinese University of Hong Kong and Florida State University has delivered the first unified convergence theory for a broad class of score based generative models in 2-Wasserstein distance, and it shows that the

Wasserstein Convergence Guarantees for Score-Based Generative Models Read More »

How Dommel and Pichler Finally Cracked the Kernel Approximation Problem That Was Holding Machine Learning Back.

How Dommel and Pichler Finally Cracked the Kernel Approximation Problem That Was Holding Machine Learning Back

Leave a Comment / Machine Learning, Optimization & Learning Theory /

How Dommel and Pichler Finally Cracked the Kernel Approximation Problem That Was Holding Machine Learning Back | AI Trend Blend AITrendBlend Machine Learning Cybersecurity Computer Vision About Statistical Learning · Journal of Machine Learning Research 26 (2025) 1–30 · 18 min read How Two Researchers from Chemnitz Quietly Fixed One of the Oldest Problems in

How Dommel and Pichler Finally Cracked the Kernel Approximation Problem That Was Holding Machine Learning Back Read More »

Sliced-Wasserstein Distances and Flows on Cartan-Hadamard Manifolds.

Sliced-Wasserstein Distances and Flows on Cartan-Hadamard Manifolds

Leave a Comment / Machine Learning, Optimization & Learning Theory /

Sliced-Wasserstein Distances and Flows on Cartan-Hadamard Manifolds | AI Trend Blend AITrendBlend Machine Learning Cybersecurity About Optimal Transport · Journal of Machine Learning Research 26 (2025) 1–76 · 18 min read Measuring Distance Between Distributions on Curved Spaces Just Got a Lot Faster Bonet, Drumetz, and Courty from ENSAE, IMT Atlantique, and Universite Bretagne Sud

Sliced-Wasserstein Distances and Flows on Cartan-Hadamard Manifolds Read More »