Deep Learning: Do-It-Yourself!

Hands-on tour to deep learning (2017)

2017 Archive, for the current version see here

Course description

Recent developments in neural network approaches (more known now as "deep learning") have dramatically changed the landscape of several research fields such as image classification, object detection, speech recognition, machine translation, self-driving cars and many more. Due its promise of leveraging large (sometimes even small) amounts of data in an end-to-end manner, i.e. train a model to extract features by itself and to learn from them, deep learning is increasingly appealing to other fields as well: medicine, time series analysis, biology, simulation.

This course is a deep dive into practical details of deep learning architectures, in which we attempt to demystify deep learning and kick start you into using it in your own field of research. During this course, you will gain a better understanding of the basis of deep learning and get familiar with its applications. We will show how to set up, train, debug and visualize your own neural network. Along the way, we will be providing practical engineering tricks for training or adapting neural networks to new tasks.

By the end of this class, you will have an overview on the deep learning landscape and its applications to traditional fields, but also some ideas for applying it to new ones. You should also be able to train a multi-million parameter deep neural network by yourself. For the implementations we will be using the PyTorch library in Python.

The topics covered in this course include:

  • Neural network approaches: feedforward networks, convolutional networks (CNNs), recurrent networks (RNNs)
  • Modern practices: backpropagation, regularization, optimization, fine-tuning
  • Deep Learning research: autoencoders, deep generative models, long short-term memory (LSTM) modules
  • CNN architectures: VGG, ResNet, fully convolutional net, multi input and multi output nets
  • RNN architectures: bidirectional RNNs, encoder-decoder sequence-to-sequence, LSTMs, GRUs

  • Schedule

    # Date Description Course Materials
    Lecture 1 Thursday
    October 5
    14h-16h
    salle Conf IV
    Course introduction
    Meet your dev environment
    First dive into CNNs
    Testing out pre-trained networks
    [forum]
    [ipynb]
    Lecture 2 Monday
    October 16
    14h-16h
    salle Conf IV
    Slides: Intro to Machine Learning
    Supervised Learning
    Unsupervised learning
    Gradient descent

    Practical: Intro to PyTorch
    Basic operations and automatic differentiation
    Linear regression
    [slides]
    [ipynb]
    Lecture 3 Thursday
    October 19
    14h-16h
    salle Conf IV
    Slides: Machine Learning basics
    Optimization
    Stochastic gradient descent
    Momentum and adaptive learning rates
    Backpropagation

    Practical: Using CNNs in pratice
    Finetuning model on new task
    Optimization
    Result visualization
    Convolutions
    [slides]
    [ipynb cats and dogs]
    [ipynb convolutions]
    Lecture 4 Thursday
    November 9
    14h-16h
    salle Conf IV
    Slides: Image classification
    K-Nearest Neighbors
    Linear classification
    Loss functions
    Regularization

    Practical: Recommender sysstems
    embeddings
    collaborative filtering
    [slides]
    [ipynb]
    Lecture 5 Thursday
    November 16
    14h-16h
    salle Conf IV
    Slides: Starting with Neural Networks
    Gradient descent
    Backpropagation
    Human-engineered features
    Neural Networks I

    Practical: RecSys with Neural Networks
    Using Fully-Connected layers
    Triplet loss
    Clustering
    Dimensionality reduction: PCA, t-SNE
    Result visualization
    [slides]
    [ipynb]
    Lecture 6 Thursday
    November 23
    14h-16h
    salle Conf IV
    Slides: Deep Neural Networks
    Neural Networks (continued)
    Convolutional Neural Networks
    Dropout
    Batch Normalization
    CNN architectures

    Practical: Sentiment analysis from text
    Embeddings
    GloVe
    1d convolutions
    [slides]
    [ipynb]
    Lecture 7 Thursday
    November 30
    14h-16h
    salle Conf IV
    Slides: Tips and tricks for training Neural Networks in practice
    Convolutions (continued)
    CNN architectures (continued)
    Visualizing and understanding CNNs
    GPUs
    Data preprocessing and augmentation
    Hyperparameters and network babysitting
    Learning rate update rules and schedules
    Common pitfalls
    Transfer learning, ensembles

    Practical: First attempts for language modeling with a simple recurrent network
    Embeddings
    Shared weights across timesteps
    Char-RNN: next character prediction
    Training and texting on Nietzsche texts
    [slides]
    [ipynb]
    Lecture 8 Thursday
    December 14
    14h-16h
    salle Conf IV
    Slides: Recurrent Neural Networks
    Vanilla RNNs, GRU, LSTM, bidirectional RNNs
    RNN architectures and setups
    Word embeddings, Language modeling, time series analysis
    Encoder-Decoder / Sequence to Sequence: machine translation
    Soft attention
    Temporal convolutions

    Practical: Autoencoders
    Linear autoencoders
    Denoising autoencoders
    [slides]
    [ipynb]
    Lecture 9 Thursday
    December 21
    14h-16h
    salle L378/L380
    Generative models. Adversarial examples and adversarial training
    Autoencoders. Variational Autoencoders
    Generative Adversarial Networks
    Adversarial examples and training
    [slides]
    [ipynb]

    Practical info

    Finding the lecture hall:

    The class will take place in room Conf IV, in the second floor of the physics department of Ecole Normale Superieure.

    GPU install

    Instructions on how to use AWS instance are on the forum.


    Instructors

  • Andrei Bursuc (Safran) - Andrei dot Bursuc at safrangroup dot com
  • Florent Krzakala (ENS) - Florent dot Krzakala at ens dot fr
  • Marc Lelarge (Safran/Inria Paris/ENS) - Marc dot Lelarge at inria dot fr

  • Enrolment

    This course is dedicated to PhD students from Doctoral School ED564: Physique en Ile de France (EDPIF).

    If you are not a student of EDPIF and wish to enroll, please get in touch with one of the instructors.

    Note that due to the limited amount of available seats, the class is currently open only for academics.


    Aknowledgements