** Deep Learning: Do-It-Yourself!**

Hands-on tour to deep learning (2017)

2017 Archive, for the current version see here

Recent developments in neural network approaches (more known now as "deep learning") have dramatically changed the landscape of several research fields such as image classification, object detection, speech recognition, machine translation, self-driving cars and many more. Due its promise of leveraging large (sometimes even small) amounts of data in an end-to-end manner, i.e. train a model to extract features by itself and to learn from them, deep learning is increasingly appealing to other fields as well: medicine, time series analysis, biology, simulation.

This course is a deep dive into practical details of deep learning architectures, in which we attempt to demystify deep learning and kick start you into using it in your own field of research. During this course, you will gain a better understanding of the basis of deep learning and get familiar with its applications. We will show how to set up, train, debug and visualize your own neural network. Along the way, we will be providing practical engineering tricks for training or adapting neural networks to new tasks.

By the end of this class, you will have an overview on the deep learning landscape and its applications to traditional fields, but also some ideas for applying it to new ones. You should also be able to train a multi-million parameter deep neural network by yourself. For the implementations we will be using the PyTorch library in Python.

The topics covered in this course include:

# |
Date |
Description |
Course Materials |

Lecture 1 | Thursday October 5 14h-16h salle Conf IV |
Course introductionMeet your dev environment First dive into CNNs Testing out pre-trained networks |
[forum] [ipynb] |

Lecture 2 | Monday October 16 14h-16h salle Conf IV |
Slides: Intro to Machine LearningSupervised Learning Unsupervised learning Gradient descent Practical: Intro to PyTorchBasic operations and automatic differentiation Linear regression |
[slides] [ipynb] |

Lecture 3 | Thursday October 19 14h-16h salle Conf IV |
Slides: Machine Learning basicsOptimization Stochastic gradient descent Momentum and adaptive learning rates Backpropagation Practical: Using CNNs in praticeFinetuning model on new task Optimization Result visualization Convolutions |
[slides] [ipynb cats and dogs] [ipynb convolutions] |

Lecture 4 | Thursday November 9 14h-16h salle Conf IV |
Slides: Image classificationK-Nearest Neighbors Linear classification Loss functions Regularization Practical: Recommender sysstemsembeddings collaborative filtering |
[slides] [ipynb] |

Lecture 5 | Thursday November 16 14h-16h salle Conf IV |
Slides: Starting with Neural NetworksGradient descent Backpropagation Human-engineered features Neural Networks I Practical: RecSys with Neural NetworksUsing Fully-Connected layers Triplet loss Clustering Dimensionality reduction: PCA, t-SNE Result visualization |
[slides] [ipynb] |

Lecture 6 | Thursday November 23 14h-16h salle Conf IV |
Slides: Deep Neural NetworksNeural Networks (continued) Convolutional Neural Networks Dropout Batch Normalization CNN architectures Practical: Sentiment analysis from textEmbeddings GloVe 1d convolutions |
[slides] [ipynb] |

Lecture 7 | Thursday November 30 14h-16h salle Conf IV |
Slides: Tips and tricks for training Neural Networks in practiceConvolutions (continued) CNN architectures (continued) Visualizing and understanding CNNs GPUs Data preprocessing and augmentation Hyperparameters and network babysitting Learning rate update rules and schedules Common pitfalls Transfer learning, ensembles Practical: First attempts for language modeling with a simple recurrent networkEmbeddings Shared weights across timesteps Char-RNN: next character prediction Training and texting on Nietzsche texts |
[slides] [ipynb] |

Lecture 8 | Thursday December 14 14h-16h salle Conf IV |
Slides: Recurrent Neural NetworksVanilla RNNs, GRU, LSTM, bidirectional RNNs RNN architectures and setups Word embeddings, Language modeling, time series analysis Encoder-Decoder / Sequence to Sequence: machine translation Soft attention Temporal convolutions Practical: AutoencodersLinear autoencoders Denoising autoencoders |
[slides] [ipynb] |

Lecture 9 | Thursday December 21 14h-16h salle L378/L380 |
Generative models. Adversarial examples and adversarial trainingAutoencoders. Variational Autoencoders Generative Adversarial Networks Adversarial examples and training |
[slides] [ipynb] |

The class will take place in room Conf IV, in the second floor of the physics department of Ecole Normale Superieure.

Instructions on how to use AWS instance are on the forum.

This course is dedicated to PhD students from Doctoral School ED564: *Physique en Ile de France* (EDPIF).

If you are not a student of EDPIF and wish to enroll, please get in touch with one of the instructors.

Note that due to the limited amount of available seats, the class is currently open only for academics.