The class will be taught in French or English, depending on attendance (all slides and class notes are in English).
Summary
Given the
sanitary situation, all classes will be online, with the following
tentative plan. Detailed class notes will be made available
2 days before each class, while connection details the night
before (I will use GotoMeeting).
Each
student is expected to read the class notes before the class.
During class, I will go over them, provide additional details and
answer questions. Classes will be held on Friday between
8.30am and 11.30am.
Date  Topics  Class
notes 
September 18 
Learning with infinite data (population setting) Decision theory (loss, risk, optimal predictors) Decomposition of excess risk into approximation and estimation errors No free lunch theorems Basic notions of concentration inequalities (MacDiarmid, Hoeffding, Bernstein) 
lecture1.pdf gotomeeting link 
September 25 
Leastsquares regression Guarantees in the fixed design settings (simple in closed form) Guarantees in the random design settings Ridge regression: dimension independent bounds 

October 2  Classical risk decomposition Approximation error Convex surrogates Estimation error through covering numbers (basic example of ellipsoids) Modern tools (no proof): Rademacher complexity, Gaussian complexity + Slepian and Lipschitz results Minimax rates (at least one proof) 

October 16 
Optimization for machine learning Gradient descent Stochastic gradient descent Generalization bounds through stochastic gradient descent 

October 23  Local averaging techniques Kernel density estimation NadarayaWatson estimators (simplest proof to be found with apparent curse of dimensionality) KNN Decision trees and associated methods 

October 30 
Kernel methods Modern analysis of nonparametric techniques (simplest proof with results depending on s and d) 

November 6 
Model selection L0 penalty with AIC L1 penalty Highdimensional estimation 

November 13 
Neural networks Approximation properties (simplest approximation result) Two layers Deep networks 

November 20 
Special topics Generalization/optimization properties of infinitely wide neural networks Double descent 
Evaluation
One
written inclass exam, and (very) simple coding assignments (to
illustrate convergence results, to be sent to
learning.theory.first.principles@gmail.com).