SOTA · Scalar-Order Theories of Deep Kernel Adaptation

The AI revolution is under way, yet we still lack a thermodynamic understanding of deep learning, that is explaining what a network can learn in the limit of large dataset and large network width.The central proposal of SOTA is that, the output of a deep network on real data can be predicted by an effective kernel - a similarity measure between data points - that shows simple low-dimensional adaptation to the dataset. This reconnects two regimes often seen as completely different: “lazy” learning (where a fixed kernel predictor explains outputs) and “rich” learning (where network features are plastic and little understood).Indeed, SOTA argues that rich learning in fully-connected networks effectively reduces predictor variance, while almost no adaptation of the mean predictor is predicted and observed. Convolutional networks instead show local adaptations of the kernel that also change the mean predictor.This is where the number of data samples is proportional to width, the classical limit in statistical physics of learning. Due to the abundance (or augmentation) of data in recent deep learning practice, also the quadratic sample-width limit will need to be investigated.Based on these consequential observations, a three-pronged investigation is proposed: - Comparison of predictions to large-scale Monte-Carlo experiments on real data. - Mechanistic explanation through a proposed mechanism of auto-ensembling. - Analysis of corrections to the effective action that are sub-leading in the proportional limit, but become relevant when data is more abundant.SOTA disrupts widely held assumptions about rich learning, and makes major progress on a long-standing challenge: Explaining the performance of deep, nonlinear networks in the feature learning regime, on real data, through a simple, effective theory.