Reading

Learning as compression: prefer the model that gives the shortest description of the data. Chapter 1 is a non-technical tour; chapter 2 makes it precise.

On this site: Kolmogorov Complexity · From Entropy to Epiplexity

Ridge regression, Newton's method, the Kalman filter, SGD, RMSprop and dropout all fall out of one natural-gradient Bayesian update.

On this site: Probability Paradigms

An illustrated map of machine learning beyond flat vector spaces: data and models with geometric, topological and algebraic structure.

A first course on diffusion and flow matching that derives correct algorithms with as little machinery as possible.

On this site: Diffusion models or Autoencoders? · DDIM vs DDPM

Statistical manifolds, Fisher information and divergences from a geometric point of view, written for information theorists.

On this site: Some common divergences · fisher-rao-ml (project)

What does it mean for two graphs to be correlated? A definition through the parameters of the random graph models behind them, applied to brain networks.

On this site: graph-corr-embedd (project)

Wrap any trained model in prediction sets with distribution-free, finite-sample guarantees. Short and practical.

Learning to predict the solutions of optimization problems: the idea behind the VAE encoder, and much of reinforcement learning.

On this site: A comprehensive list of different types of VAEs

V-information: what a computationally bounded observer can actually extract. Unlike Shannon information, it can be created by computation.

On this site: From Entropy to Epiplexity

Yes, up to a small error bounded uniformly in time: softmax attention can implement kernel smoothing, which tracks the Kalman filter.

On this site: Implementing the Kalman Filter in Python · kalman-transformer (project)

Intermediate layers of LLMs often give better representations than the last one, for Transformers and state-space models alike.

On this site: The Platonic Representation Hypothesis: Are AI Models Converging on Universal Representations?

Diffusion for data in bounded ranges, using multiplicative beta-distributed transitions instead of additive Gaussian noise.

Hyper-connections widen the residual stream but break its identity mapping; mHC restores it by constraining the connections to a manifold.

Real-world data has low Kolmogorov complexity and neural networks share that bias toward simplicity, so no-free-lunch theorems bite less than they seem.

On this site: Kolmogorov Complexity

Both directions: what diffusion models learn as representations, and how representations are used to improve diffusion.

On this site: Diffusion models or Autoencoders?

The cold posterior effect: in deep nets, a deliberately sharpened posterior predicts better than the actual Bayes posterior.

On this site: Probability Paradigms

The standard introduction to variational inference: posit a family, minimize the KL to the posterior, and scale it with stochastic optimization.

On this site: A comprehensive list of different types of VAEs

The Transformer: attention only, no recurrence or convolutions.

On this site: Implementing the Transformer in Python

The spectral density of a graph's adjacency matrix as a model-agnostic tool to fit and select random graph models.

On this site: logit-graph (project)

A compact survey of reinforcement learning that treats TD learning and Q-learning as stochastic approximation.

On this site: Simple Reinforcement Learning Example

Intelligence as skill-acquisition efficiency rather than skill, with the ARC benchmark to measure it.

Late in training, last-layer features collapse to their class means, which form a simplex equiangular tight frame. A review of why, and what it implies.

On this site: What is Neural Collapse? A Simpler Look

Instead of denoising samples, update the parameters of simple distributions by Bayesian inference. No forward process, and discrete data works natively.

LLaDA: a language model trained from scratch as a masked diffusion model rather than autoregressively, and competitive at 8B scale.

Drifting models move the generated distribution during training instead of at inference time, so sampling takes a single step.

An entropy for networks from their spectrum, borrowed from quantum statistical mechanics, which yields a Jensen-Shannon distance between networks.

Node embeddings from biased random walks that interpolate between breadth-first and depth-first exploration.

Learn to sample objects like sets and graphs with probability proportional to a reward, amortizing work that MCMC would otherwise do.

Kalman filters with neural-network transitions and emissions, learned with the variational machinery of VAEs.

On this site: Implementing the Kalman Filter in Python · A comprehensive list of different types of VAEs

How DeepSeek-V3 was co-designed with its hardware: Multi-head Latent Attention, mixture of experts, FP8 training and the interconnect limits behind them.

On this site: Deepseek, an overview and quick notes · Mixture of Experts

Train continuous normalizing flows without simulation, by regressing vector fields along fixed conditional paths; diffusion paths are a special case.

On this site: Flow Matching, a short overview · flow-match (project)

Dense networks contain sparse subnetworks that, reset to their original initialization, train to the same accuracy.

On this site: The Lottery Ticket Hypothesis

Message passing where each node learns how much weight to give each neighbor, via masked self-attention.

On this site: Graph Convolutional Networks

Learnable activation functions on the edges instead of fixed ones on the nodes, inspired by the Kolmogorov-Arnold representation theorem.

On this site: Kolmogorov-Arnold Networks