The class will be taught in French or English, depending on attendance (all slides and class notes are in English).
Summary
All classes will be "in real life" at ENS (rue d'Ulm), on Friday between 9am and 12pm.
The class
will follow the book in preparation (draft available here).
Each
student will benefit more from the class is the corresponding
sections are read before class.
Date | Topics | Book chapters Figures to reproduce |
October 8 Salle des Résistants |
Learning with infinite data (population setting) -Decision theory (loss, risk, optimal predictors) -Decomposition of excess risk into approximation and estimation errors -No free lunch theorems -Basic notions of concentration inequalities (MacDiarmid, Hoeffding, Bernstein) |
Chapter 2 |
October 15 Salle Favard (46, rue d'Ulm) |
Liner Least-squares regression -Guarantees in the fixed design settings (simple in closed-form) -Ridge regression: dimension independent bounds -Guarantees in the random design settings -Lower bound of performance |
Chapter 3 Board Figures: 3.1, 3.2 |
November 5 Salle Dussane |
Empirical risk minimization -Convexification of the risk -Risk decomposition -Estimation error: finite number of hypotheses and covering numbers -Rademacher complexity -Penalized problems |
Chapter 4 Board |
November 12 Salle Dussane |
Optimization for machine learning -Gradient descent -Stochastic gradient descent -Generalization bounds through stochastic gradient descent |
Chapter 5 Board Figures: p. 90, p. 108 |
November 19 Salle Favard (46, rue d'Ulm) |
Local averaging techniques -Partition estimators -Nadaraya-Watson estimators -K-nearest-neighbors -Universal consistency |
Chapter 6 Board Figures: 6.2, 6.3, 6.4 |
November 26 Salle Favard (46, rue d'Ulm) |
Kernel methods -Kernels and representer theorems -Algorithms -Analysis of well-specified models -Sharp analysis of ridge regression -Universal consistency |
Chapter 7 Board |
December 3 Salle Dussane |
Model selection -L0 penalty -L1 penalty -High-dimensional estimation |
Chapter 8 Board |
December 10 Salle Dussane |
Neural networks -Single hidden layer neural networks - Estimation error - Approximation properties and universality |
Chapter 9 Board |
December 17 Emmy NOETHER (U / V), 45 rue d'Ulm |
Exam |
Evaluation
One
written in-class exam, and (very) simple coding assignments (to
illustrate convergence results, to be sent to
learning.theory.first.principles@gmail.com). For all classes, the
coding assignment is to reproduce the experiments shown in the book
draf and send only the figures to the address above.