{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# TP2: Autoencoder - Learning to choose the best hyperparameters" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "An autoencoder (**AE**) is a special type of neural network that is trained to copy its input to its output. For example, given an image of a handwritten digit, an autoencoder first encodes the image into a lower dimensional latent representation, then decodes the latent representation back to an image. An autoencoder learns to compress the data while minimizing the reconstruction error. This is an example of unsupervised learning. Autoencoder can be used to learn representation (the by product of the encoder), and to detect anomalies. \n", "\n", "An autoencoder consists in two blocks: \n", "- the encoder projects the input in a latent space of \"reduced\" dimension (in comparison with the input). \n", "- the decoder maps some latent representation back in the input space. \n", "\n", "Basically, if you consider an input $\\mathbf{x}$ : \n", "$$\\mathbf{z} = encoder(\\mathbf{x})$$\n", "$$\\mathbf{\\tilde{x}} = decoder(\\mathbf{z})$$\n", "\n", "The goal is to train the AE, the goal is to minimize the reconstruction error. This error can be defined as the mean square error for instance : \n", "$\\min\\Vert\\mathbf{\\tilde{x}}-\\mathbf{x}\\Vert^2$\n", "\n", "For this lab session, you can work with the MNIST dataset and we will train 3 different models:\n", "- A linear Endoder/Decoder\n", "- A non-linear Fully Connected Encoder/Decoder\n", "- A Convolutional Encoder/Decoder\n", "\n", "More details and constraints will be given through the practical session. \n", "Program:\n", "1. Import Data and visualize the images \n", "2. Build a train/test wrapper\n", "3. Train a linear model and understand how the latent dimension affects the reconstruction\n", "4. Use the model to denoise\n", "5. Use the model for anomaly detection\n", "6. Train and use a Fully Connected model for denoising and anomaly detection\n", "7. Train and use a Convolutional model for denoising and anomaly detection\n", "\n", "\n", "First, we import the package required for the TP." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import numpy as np # Library for numerical calculations\n", "import matplotlib.pyplot as plt # Library for plotting\n", "\n", "import math # Library for mathematical functions\n", "\n", "import torch # Library for tensor computations - for Deep Learning\n", "import torch.nn as nn # Neural Network module\n", "import torch.optim as optim # Optimization module\n", "from torchvision import datasets, transforms # Computer Vision module\n", "\n", "%matplotlib inline\n", "%config InlineBackend.figure_formats=['retina']\n", "%config InlineBackend.figure_format = 'retina'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Functions that will be useful to:\n", "- Plot the reconstruction\n", "- Plot the data in the latent space\n", "- Print the number of parameters" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "\n", "def plot_result(model):\n", " '''plot_result\n", " Plot the first 5 images of the test set and their reconstructions\n", " Input: \n", " - model: the trained model\n", " Output:\n", " - None\n", " '''\n", " first_outputs = model(first_batch_images)\n", " plt.figure(figsize=(10, 4))\n", " for i in range(5):\n", " plt.subplot(2, 5, i+1)\n", " plt.imshow(first_batch_images[i].cpu().numpy().reshape(28, 28), cmap='gray')\n", " plt.title('Input')\n", " plt.xticks([])\n", " plt.yticks([])\n", " plt.subplot(2, 5, i+6)\n", " plt.imshow(first_outputs[i].cpu().detach().numpy().reshape(28, 28), cmap='gray')\n", " plt.title('Output')\n", " plt.xticks([])\n", " plt.yticks([])\n", " plt.show()\n", "\n", "def plotLatentScatter(encoder, classes=None):\n", " '''plotLatentScatter\n", " Plot the test dataset in the latent space of the encoder\n", " Input:\n", " - encoder: the encoder model\n", " - classes: the classes to plot\n", " Output:\n", " - None\n", " '''\n", " latent_numpy = []\n", " labels_numpy = []\n", " for images, labels in test_loader:\n", " x = images\n", " latent_numpy.append(encoder(x).cpu().detach().numpy())\n", " labels_numpy.append(labels.numpy())\n", " if classes is None : \n", " classes = range(10)\n", " latent_numpy = np.concatenate(latent_numpy)\n", " labels_numpy = np.concatenate(labels_numpy)\n", " for c in classes:\n", " plt.scatter(latent_numpy[labels_numpy==c,0],latent_numpy[labels_numpy==c,1],marker=\"o\",alpha=0.1,label=str(c))\n", " plt.legend()\n", " plt.show()\n", "\n", "def print_nparameters(model):\n", " '''print_nparameters\n", " Print the number of parameters of the model\n", " Input:\n", " - model: the model\n", " Output:\n", " - None\n", " '''\n", " nparameters = 0\n", " for parameter in model.parameters():\n", " nparameters += parameter.numel()\n", " print('Number of parameters:', nparameters)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 1 Import Data and visualize the images " ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Train dataset: Dataset MNIST\n", " Number of datapoints: 60000\n", " Root location: \n", " Split: Train\n", " StandardTransform\n", "Transform: ToTensor() \n", "\n", "Test dataset: Dataset MNIST\n", " Number of datapoints: 10000\n", " Root location: \n", " Split: Test\n", " StandardTransform\n", "Transform: ToTensor()\n" ] } ], "source": [ "train_dataset = datasets.MNIST(root='', train=True, transform=transforms.ToTensor(), download=True)\n", "test_dataset = datasets.MNIST(root='', train=False, transform=transforms.ToTensor(), download=True)\n", "\n", "print('Train dataset:', train_dataset, \"\\n\")\n", "print('Test dataset:', test_dataset)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAABmcAAAKDCAYAAADvvt4HAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8ekN5oAAAACXBIWXMAAB7CAAAewgFu0HU+AABDEElEQVR4nO3dCXRV1d034BMIs4oyiDiBQ8XKIIJYBxSrFC2CE1D0tc5LbR1qq6JflYpVUYvW93W2ap1daisOqDgVERxwwKkqQxERilIFBCEMhZB8694uUqzQs5GTndyb51krK5fkx3/vq3BI8rvnnJLKysrKBAAAAAAAgCjqxVkGAAAAAACAHOUMAAAAAABARMoZAAAAAACAiJQzAAAAAAAAESlnAAAAAAAAIlLOAAAAAAAARKScAQAAAAAAiEg5AwAAAAAAEJFyBgAAAAAAICLlDAAAAAAAQETKGQAAAAAAgIiUMwAAAAAAABEpZwAAAAAAACJSzgAAAAAAAESknAEAAAAAAIhIOQMAAAAAABCRcgYAAAAAACCi0piL1XbLly9PPvjgg/zj1q1bJ6Wl/vMA31ZeXp7MnTs3/7hz585J48aNk2LgGAjU1WOg4x8QyjEQqKuK8fiX4xgI1OQx0BFnDbmD8R577FHT2wAKyJtvvpn06NEjKQaOgUBdPQY6/gHfhWMgUFcVy/EvxzEQqMljoMuaAQAAAAAARKScWUPu9EWAunrcKKbnAsRRLMeNYnkeQFzFcuwolucBxFNMx41iei5A4R03lDNrcF1JoC4fN4rpuQBxFMtxo1ieBxBXsRw7iuV5APEU03GjmJ4LUHjHDeUMAAAAAABARMoZAAAAAACAiJQzAAAAAAAAESlnAAAAAAAAIlLOAAAAAAAARKScAQAAAAAAiEg5AwAAAAAAEJFyBgAAAAAAICLlDAAAAAAAQETKGQAAAAAAgIiUMwAAAAAAABEpZwAAAAAAACJSzgAAAAAAAESknAEAAAAAAIhIOQMAAAAAABCRcgYAAAAAACAi5QwAAAAAAEBEyhkAAAAAAICIlDMAAAAAAAARKWcAAAAAAAAiUs4AAAAAAABEpJwBAAAAAACISDkDAAAAAAAQkXIGAAAAAAAgIuUMAAAAAABARMoZAAAAAACAiJQzAAAAAAAAESlnAAAAAAAAIlLOAAAAAAAARKScAQAAAAAAiEg5AwAAAAAAEJFyBgAAAAAAICLlDAAAAAAAQETKGQAAAAAAgIhKYy4GABS37t27p2bOPPPMoFnHHXdcaubee+8NmnXDDTekZt55552gWQAAAAAbypkzAAAAAAAAESlnAAAAAAAAIlLOAAAAAAAARKScAQAAAAAAiEg5AwAAAAAAEJFyBgAAAAAAICLlDAAAAAAAQETKGQAAAAAAgIiUMwAAAAAAABGVxlwMYqtfv35qpnnz5klsZ555ZlCuadOmqZkOHToEzTrjjDNSM9dcc03QrKOPPjoot3z58tTMVVddFTTrt7/9bVAOqB5du3YNyr3wwgupmU022SRoVmVlZWrm2GOPDZp16KGHpmZatmwZNAug2Bx44IFBuQceeCAo16tXr9TM1KlTg2YBrM3QoUMz+z6yXr2w1y3vv//+Qblx48YF5QDAmTMAAAAAAAARKWcAAAAAAAAiUs4AAAAAAABEpJwBAAAAAACISDkDAAAAAAAQkXIGAAAAAAAgIuUMAAAAAABARMoZAAAAAACAiEpjLkbx2nbbbVMzDRs2DJq19957p2Z69uwZNGvTTTdNzQwYMCApZLNnzw7KXX/99amZI444ImjW4sWLg3Lvv/9+ambcuHFBs4DqscceewTlRo4cGZRr3rx5aqaysjKzY82KFSuCZrVs2TI1s+eeewbNeuedd4JyoXuDNPvtt19mf84fe+yxDHZEsenRo0dQ7q233qr2vQCccMIJqZkLLrggaFZFRUWSldCvYQEglDNnAAAAAAAAIlLOAAAAAAAARKScAQAAAAAAiEg5AwAAAAAAEJFyBgAAAAAAICLlDAAAAAAAQETKGQAAAAAAgIiUMwAAAAAAABEpZwAAAAAAACIqjbkYhadr165BuRdffDE107x58wx2VLdUVFSkZoYOHRo0q6ysLDXzwAMPBM2aM2dOUG7BggWpmalTpwbNAv6tadOmQblu3bqlZu6///6gWW3btk1imzZtWmpmxIgRQbMeeuih1Myrr74aNCv0uHvllVcG5SDN/vvvH5T73ve+l5p57LHHMtgRhaRevfTX42233XZBs9q1axeUKykpCcoBfNdjTePGjaPsBSg8P/jBD1IzP/3pT4Nm9erVKyjXsWPHJCvnnXdeaubzzz8PmtWzZ8/MfibwxhtvBOVYP86cAQAAAAAAiEg5AwAAAAAAEJFyBgAAAAAAICLlDAAAAAAAQETKGQAAAAAAgIiUMwAAAAAAABEpZwAAAAAAACJSzgAAAAAAAESknAEAAAAAAIioNOZiFJ5Zs2YF5ebPn5+aad68eVLI3njjjaDcwoULUzM//OEPg2atWLEiNXPfffcFzQKKxx/+8Ieg3NFHH50Usm7duqVmNtpoo6BZ48aNS83sv//+QbO6dOkSlIOsHHfccUG5CRMmVPteKDxt27ZNzZxyyilBs+6///6g3JQpU4JyQN3Su3fvoNxZZ52V2Zohx6N+/foFzfriiy8y2BHwXQ0ePDgod91116VmWrVqFTSrpKQkKPfSSy+lZlq3bh006+qrr06yErL/0H0dddRRGeyI/+TMGQAAAAAAgIiUMwAAAAAAABEpZwAAAAAAACJSzgAAAAAAAESknAEAAAAAAIhIOQMAAAAAABCRcgYAAAAAACAi5QwAAAAAAEBEpTEXo/B89dVXQbkhQ4akZvr16xc06913303NXH/99UlW3nvvvaDcj370o6DckiVLUjMdO3YMmnX22WcH5YDi0b1799TMIYccEjSrpKQkycq4ceOCck8++WRq5pprrgma9fnnn2fyb0bOggULUjMHHHBA9P+uEKJePa+n4ru74447Mps1bdq0zGYBxaVnz56pmbvuuitoVvPmzZOsXH311amZmTNnZrYe8E2lpWE/et59991TM7fffnvQrKZNm6Zmxo8fHzTrsssuC8q98sorqZlGjRoFzfrTn/6UmunTp0+SlYkTJ2Y2i/XnOz0AAAAAAICIlDMAAAAAAAARKWcAAAAAAAAiUs4AAAAAAABEpJwBAAAAAACIqDSpQWVlZcnIkSOTCRMmJJMnT04WLlyY/1hlZWXwjJKSkmT69OnVuk8AAAAAAICCLmdWrVqVDB8+PLn66quTpUuXfuNzocVMrpTJZXPvAQAAAAAACkX0cqa8vDw54ogjktGjR1cVMauLlvWxvnkAAAAAAIA6Wc787ne/S55++ulvlDINGjRIOnfunLRv3z5p1qyZs2EAAAAAAICiFbWcWbFiRTJixIhvlDLDhg1LTj/99KR58+Yxt0LGHn/88dTMiy++GDRr8eLFqZldd901aNbJJ5+cmrnmmmuCZi1ZsiTJykcffRSUO/XUUzNbE6hZXbt2Dcq98MILqZlNNtkks7NMn3nmmaBZRx99dFCuV69eqZmhQ4cGzbrjjjtSM3Pnzg2a9f7776dmKioqgmYdcsghQblu3bqlZt55552gWRSvLl26pGbatGkTZS8Upyy/zwr5Nwqom44//vjUzJZbbpnZei+99FJQ7t57781sTWD9/fSnP83se78sv14ZPHhw0KxFixZlsKP1W7NPnz6ZrTl79uzUzD333JPZetTycubll1+u+sF7rqC5//77k4EDB8bcAgAAAAAAQI2qF3OxadOmVRUz3bt3V8wAAAAAAAB1TtRyZuHChVWP991335hLAwAAAAAA1L1yZvPNN696vPHGG8dcGgAAAAAAoO6VM9tvv33V4y+++CLm0gAAAAAAAHWvnMldymyLLbbIPx43blzMpQEAAAAAAOpeOVO/fv3kjDPOSCorK5OpU6cmo0ePjrk8AAAAAABA3Spncn79618nPXv2zBc0J510UjJt2rTYWwAAAAAAAKgxpbEXrFevXjJq1Khk0KBByZgxY5IePXokl19+eb6oadq0aeztENGiRYsym/X1119nNuuUU04Jyj388MNBuYqKig3cEVBodtppp9TMkCFDgmY1b948NTNv3rygWXPmzEnN3HPPPUGzysrKgnJPP/10JpnarEmTJkG5c889NzVzzDHHZLAjClnfvn0z+zNH3dKmTZug3HbbbZfZmp999llms4DC0KpVq6Bc7mc6WX2vvHDhwtRM7udIQM267LLLUjMXXnhh0Kzci/jT3HzzzUGzhg4dGvVnlKEuuuii6Gv+4he/SM3MnTs3yl6oJeXM6n+wc/eeadiwYf4vw9lnn52cf/75Sbdu3ZKtttoqadasWfC8kpKS5I9//GM17hgAAAAAAKCAy5m77747X6islnuca0eXL1+eTJgwYb1m5X6fcgYAAAAAACgk0cuZ/yxWctYsa1Z/Ls1//h4AAAAAAIBCEL2c2XbbbRUrAAAAAABAnRW9nPn0009jLwkAAAAAAFBr1KvpDQAAAAAAANQlyhkAAAAAAICIlDMAAAAAAAARKWcAAAAAAAAiKo25GGTlkksuCcp17949NdOrV6+gWb179w7KPf/880E5oPZr1KhRUO6aa65JzfTt2zdo1uLFi1Mzxx13XNCsiRMnpmaaNGkSNIv1t+2229b0FigAHTp0yGzWRx99lNksar+Qf3ty2rRpk5r529/+ltm/UUBhaN++fVBu5MiRSWw33HBDambs2LFR9gJ10cUXXxyUu/DCC1MzK1asCJr13HPPpWYuuOCCoFnLli1LstK4ceOgXJ8+fTL7/rCkpCQ1c/nllwfNeuKJJ4Jy1BxnzgAAAAAAABTqmTP169f/RstXXl7+XzNZWNc6AAAAAAAARV/OVFZW5suS3PsNyQAAAAAAABSrzC9rFlK6KGYAAAAAAIC6KtMzZ4YNG5ZJBgAAAAAAoFgpZwAAAAAAAAr5smYAAAAAAACsm3IGAAAAAACgUC9rBrEsWbIkKHfKKaekZt55552gWbfffntQbuzYsamZiRMnBs266aabUjOVlZVBs4D1t9tuuwXl+vbtm9mahx12WGpm3Lhxma0HFI+33nqrprdQp22yySapmYMPPjho1k9/+tPUTJ8+fZKsXHbZZUG5hQsXZrYmULNCj0ddunTJbM0xY8YE5a677rrM1gT+bdNNNw3KnX766UG5kJ9HPffcc0GzDj/88CSmHXfcMSj3wAMPBOW6d++eZOWRRx5JzYwYMSKz9ahZzpwBAAAAAACISDkDAAAAAABQFy9rNmXKlGT27NnJggULkuXLl6/XpZqOO+64at0bAAAAAABAUZQzueti/9///V/yzDPPJF9//fV3nqOcAQAAAAAACkWNlDPl5eXJkCFDkuuvvz7/67WdJVNSUrLOz63+fO5zq3MAAAAAAACFoEbKmZ/97GfJXXfd9a3iZc2iJa2wWZ/LngEAAAAAANQW9WIv+OSTTyZ33nlnVdnSrVu35M9//nMyefLkpF69elWly4wZM5IPPvgg+ctf/pJcddVVSffu3atKmY022ii57bbb8plPPvkk9lMAAAAAAAAonHImV7SstvfeeyevvfZaMmDAgKRDhw7fyLVr1y7p2LFjcsABByTnn39+/v40Tz/9dLLVVlslZWVlyemnn568+uqr+RwAAAAAAEChiFrOzJs3L5kwYULVr2+99dakYcOGwb//xz/+cfLKK68k2267bf6+NSeddFLy17/+tZp2CwAAAAAAUOD3nFldzOQuZ7brrrvmz4xZX7kzZW6//fbkoIMOSlauXJlccsklyaOPPloNu6UYTJ8+PTVzwgknBM3K3ScpxLHHHptJJqdZs2apmXvvvTdo1pw5c4JywL9de+21Qbk175m2LuPGjQuaFZoje7nLq4aoqKio9r3Ad9GiRYukNsp93Z/VsbR3795Bs7beeuvUTOiLxI455pjMjiHLli0LmvXGG2+kZv75z38GzSotTf+W7+233w6aBRSGww8/fL2uapKF3Atp0xx//PFBs77++usMdgR81699WrVqldmav/jFL4Jym2++eWrmxBNPDJp16KGHpmY6deoUNCt3a40QIfdGD71/+v3335+aWbJkSdAsar+oZ858/vnnVY933333/5pdsWLFOj/3ox/9KF/s5P5Qjx49Olm4cGGm+wQAAAAAACiKcmbBggVVj9u0afOtzzdq1Cj4VWW9evXKv8+dPZO79wwAAAAAAEAhiFrO1K9f/7+eXr/JJpus9SybtFPs0rIAAAAAAAB1spzZbLPNqh4vXrz4v15fcOrUqcHXIJ0/f35mewQAAAAAACiacmbHHXesevzZZ5996/OdO3euejxmzJj/Omv8+PFVjzfeeOPM9ggAAAAAAFA05UynTp2qHk+aNOlbn993333z7ysrK5P7778/+eKLL9Y657HHHkvefffdql9///vfr5b9AgAAAAAAFHQ5k7tPTMeOHfPlS66cWbhw4Tc+P2jQoKRhw4ZJSUlJ/rJlvXv3Tl599dWqzy9fvjy59dZbk2OPPTafydl0002TffbZJ+bTAAAAAAAAKIxyJufHP/5x/n1FRUXy3HPPfeueNL/85S/z5U2ufPnoo4+S/fbbL9lkk02SbbbZJl/EnHHGGcnSpUurMr/61a+SRo0axX4aAAAAAAAAhVHODB48OP8+V6788Y9//Nbnhw0blvTs2bOqfMm9Lysry9+jZsWKFVUfzznwwAOTCy+8MPZTAAAAAAAA+M5Kk8i6d++eLFiwIP94dcmypiZNmiTPPvtsct555yW33357smrVqm9lGjRokD+D5qqrrkrq1YveL1FkcvcwCjFt2rSg3LXXXpuayRWLIa644orUTLt27YJmDR8+PDWTK0GhLujXr19QrmvXrkG53AsH0owaNSpoFjUnd1ZvVv+/c957770N3BF1wbJlyzL7M5e7/G+amnhhU5cuXYJya/ve4D+Vl5cHzcqdaZ9mbffAXJs777wzKDdx4sTUzLhx44Jmrevem2uaPXt20Kzc91dppkyZEjQLqFnt27cPyo0cOTKJ7ZNPPsnk2AZUn9yL3kPMnTs3KNe6devUzIwZM4JmhX69m5XPP/88KLdo0aKgXNu2bVMz8+bNC5r15JNPBuUoDtHLmZzmzZv/1883bdo0ufnmm5OLL744GT16dDJ16tT8/Wk23njj/D1r+vbtm7Rp0ybafgEAAAAAAAq6nAm1xRZbJCeddFJNbwMAAAAAACAzrgkGAAAAAABQrOXMgw8+mLRo0SL/tsMOOwRfMxoAAAAAAKBYRC1nPv300/y9Y77++utkzz33TEpLa/VV1QAAAAAAAAq7nNloo42qHrdv3z7m0gAAAAAAAHWvnGnbtm3V41WrVsVcGgAAAAAAoO6VM7vuumvV448//jjm0gAAAAAAALVC1Ju+fO9730u6d++evP3228lf/vKXpKys7BuXOoPa7MMPPwzK/eQnP0nN9O/fP2jWXXfdlZo57bTTgv/+pfnRj34UNAsKXZMmTYJyDRs2DMp9+eWXqZmHH344aBbrp1GjRkG5Sy65JLM1X3zxxaDcr3/968zWpHidfvrpqZmZM2cGzdp7772T2mjWrFlBuccffzw1M3ny5KBZr7/+elLITj311NRM69atg2Z98sknGewIqA0uuOCCoFxFRUUS21VXXRV9TWD95O4DHuLwww8Pyj311FOpmRYtWgTNmj59emrmiSeeCJp19913p2a++uqroFkPPfTQel8takNnUbdEPXMmZ+jQofn3ixcvTi666KLYywMAAAAAANStcuawww5LzjjjjKSysjK58cYb86/8WLlyZextAAAAAAAA1I1yJueGG25Ifve73yX16tVLrrnmmmTnnXdOhg8fnrz66qvJ/Pnzk1WrVtXEtgAAAAAAAArvnjPbb799/n2DBg2SqVOnfuvz9evX/8avc2fQzJgxI7n44ou/03olJSVJeXn5d9wtAAAAAABAgZczn3766b8Gl659dK6MWbNYyb3958cBAAAAAACKVeblzPpQyAAAAAAAAHVN9HJm2LBhsZcEAAAAAACoNZQzAAAAAAAAEdWLuRgAAAAAAEBdV6P3nIFitHDhwtTMfffdFzTrjjvuSM2Ulob9Nd5vv/1SM/vvv3/QrJdeeikoB3XFP//5z9TMnDlzouylmDRq1Cg1M3To0KBZQ4YMSc3Mnj07aNbvf//7oFxZWVlQDtL87ne/q+ktENmBBx6Y2ayRI0dmNguoPl27dk3N9OnTJ4ntiSeeCMpNnTq12vcCxPHGG28E5Vq3bp0UqpCfkeX06tUrKFdRUZGa+eSTT4JmUbc4cwYAAAAAACAi5QwAAAAAAEBEyhkAAAAAAIBCv+dMSUlJsmrVquSAAw6ojvHfWmvMmDHVvg4AAAAAAECtLWdyKisrk3HjxlXX+Ko1cuUMAAAAAABAUtcva6Y0AQAAAAAAiHzmDAAAAAAAABHKmVwxU1pamkybNq06xgMAAAAAABSsajtzJqddu3bVOR6i6tKlS1Bu4MCBqZkePXoEzcqVnFmZNGlSamb8+PGZrQd1yahRo2p6CwWla9euQbkhQ4akZgYPHhw064knnkjNDBgwIGgWQCF57LHHanoLQIDnn38+NbPZZptltt7rr78elDvhhBMyWxOgtmjSpElQrqKiIrMrSD300ENBs6hbqu2eMwAAAAAAAHybcgYAAAAAACAi5QwAAAAAAEBEyhkAAAAAAICIlDMAAAAAAAARKWcAAAAAAAAKvZwpKSmpjrEAAAAAAAAFr1rKmcrKyuoYCwAAAAAAUPBKsx44Y8aM/HtnzwAAAAAAAEQoZ9q1a5f1SPjOOnTokJo588wzg2YdeeSRQbktttgiiWnVqlVBuTlz5qRmKioqMtgR1H6hLyAIzR1++OGpmbPPPjupC371q1+lZn7zm98EzWrevHlq5oEHHgiaddxxxwXlAABqQsuWLaN+v3bzzTcH5crKyjJbE6C2eO6552p6C1B9lzUDAAAAAABg7ZQzAAAAAAAAESlnAAAAAAAAIlLOAAAAAAAARKScAQAAAAAAiEg5AwAAAAAAEJFyBgAAAAAAICLlDAAAAAAAQESlMReDEFtssUVq5uijjw6adeaZZ6Zm2rdvn9RWEydOTM0MHz48aNaoUaMy2BEUh8rKykxzIcet66+/PmjWnXfemZqZP39+0Kw999wzNXPssccGzdp1112DcltvvXVqZtasWUGznnvuudTMzTffHDQLoNiUlJQE5XbaaafUzOuvv57BjoC1ueuuu4Jy9erFfe3sa6+9FnU9gNrkoIMOquktQJ4zZwAAAAAAACJSzgAAAAAAAESknAEAAAAAAIhIOQMAAAAAABCRcgYAAAAAACAi5QwAAAAAAEBEyhkAAAAAAICIlDMAAAAAAAARKWcAAAAAAAAiKo25GMWrTZs2qZlddtklaNaNN96Ymtl5552T2uqNN95IzVx99dVBs5544onUTEVFRdAsoPrUr18/NXP66acHzRowYEBqZtGiRUGzvve97yWxvfbaa6mZsWPHBs26+OKLM9gRQHGqrKwMytWr5/V4UF26du2amundu3fQrJDv61asWBE066abbkrNfPHFF0GzAIrR9ttvX9NbgDxfqQMAAAAAAESknAEAAAAAAIhIOQMAAAAAABCRcgYAAAAAACAi5QwAAAAAAEBEyhkAAAAAAICIlDMAAAAAAAARKWcAAAAAAAAiUs4AAAAAAABEVBpzMWqPFi1aBOX+8Ic/BOW6du2amtl+++2T2ui1114Lyv3+978Pyj333HOpmWXLlgXNAqrHhAkTgnJvvfVWUK5Hjx5JVrbYYovUTJs2bTJbb/78+UG5hx56KCh39tlnb+COAMjSXnvtlZq5++67o+wFis2mm26aydd2oT777LOg3HnnnZfZmgDF6OWXXw7K1asXdl5DRUXFBu6IusqZMwAAAAAAABEpZwAAAAAAACJSzgAAAAAAAESknAEAAAAAAIhIOQMAAAAAABCRcgYAAAAAACAi5QwAAAAAAEBEyhkAAAAAAICISmMuxob5wQ9+EJQbMmRIamaPPfYImrXVVlsltdHSpUuDctdff31q5oorrgiatWTJkqAcUPvNnj07KHfkkUcG5U477bTUzNChQ5PYrrvuutTMLbfcEjTr448/zmBHAGSlpKSkprcAAFCQPvzww6DctGnTgnLbb799amaHHXYImjV37tygHMXBmTMAAAAAAAARKWcAAAAAAAAiUs4AAAAAAABEpJwBAAAAAACISDkDAAAAAAAQkXIGAAAAAAAgIuUMAAAAAABARMoZAAAAAACAiJQzAAAAAAAAEZXGXIwNc8QRR2Say9KkSZNSM0899VTQrPLy8tTM73//+6BZCxcuDMoBrM2cOXOCcpdcckkmGQDIeeaZZ1IzgwYNirIXYN2mTJmSmnnttdeCZvXs2TODHQGQpSuuuCIod8cdd6Rmhg8fHjTrrLPOyuTnsBQGZ84AAAAAAABEpJwBAAAAAACISDkDAAAAAAAQkXIGAAAAAAAgIuUMAAAAAABARMoZAAAAAACAiJQzAAAAAAAAESlnAAAAAAAAIiqprKysjLlgbTZ79uxkm222qeltAAXk73//e7L11lsnxcAxEKirx0DHP+C7cAwE6qpiOf7lOAby32yyySZBuT/96U+pmd69ewfNevTRR1MzJ554YtCsJUuWBOWouWOgM2cAAAAAAAAiUs4AAAAAAABEpJwBAAAAAACISDkDAAAAAAAQkXIGAAAAAAAgIuUMAAAAAABARMoZAAAAAACAiJQzAAAAAAAAESlnAAAAAAAAIiqNuRgAAAAAANR2ixYtCsr95Cc/Sc0MHz48aNbPf/7z1Mwll1wSNGvSpElBOWqOM2cAAAAAAAAiUs4AAAAAAABEpJwBAAAAAACISDkDAAAAAAAQkXIGAAAAAAAgIuUMAAAAAABARMoZAAAAAACAiJQzAAAAAAAAEZXGXAwAAAAAAIrFokWLUjNnnXVW0KzQHMXBmTMAAAAAAAARKWcAAAAAAAAiUs4AAAAAAABEpJwBAAAAAACISDkDAAAAAAAQkXIGAAAAAAAgIuUMAAAAAABARMqZNZSXl9f0FoACU0zHjWJ6LkAcxXLcKJbnAcRVLMeOYnkeQDzFdNwopucCFN5xQzmzhrlz59b0FoACU0zHjWJ6LkAcxXLcKJbnAcRVLMeOYnkeQDzFdNwopucCFN5xQzkDAAAAAAAQUUllZWVlzAVrs+XLlycffPBB/nHr1q2T0tLSmt4SUEtPX1zdknfu3Dlp3LhxUgwcA4G6egx0/ANCOQYCdVUxHv9yHAOBmjwGKmcAAAAAAAAiclkzAAAAAACAiJQzAAAAAAAAESlnAAAAAAAAIlLOAAAAAAAARKScAQAAAAAAiEg5AwAAAAAAEJFyBgAAAAAAICLlDAAAAAAAQETKGQAAAAAAgIiUMwAAAAAAABEpZwAAAAAAACJSzgAAAAAAAESknAEAAAAAAIhIOQMAAAAAABCRcgYAAAAAACAi5QwAAAAAAEBEyhkAAAAAAICIlDMAAAAAAAARKWcAAAAAAAAiUs4AAAAAAABEpJwBAAAAAACISDkDAAAAAAAQkXIGAAAAAAAgIuUMAAAAAABARMoZAAAAAACAiJQzAAAAAAAAESlnAAAAAAAAIlLOAAAAAAAARKScAQAAAAAAiEg5AwAAAAAAEJFyBgAAAAAAICLlDAAAAAAAQETKGQAAAAAAgIiUMwAAAAAAABEpZwAAAAAAACJSzgAAAAAAAESknAEAAAAAAIhIOQMAAAAAABCRcgYAAAAAACAi5QwAAAAAAEBEyhkAAAAAAICIlDMAAAAAAAARKWcAAAAAAAAiUs4AAAAAAABEpJwBAAAAAACISDkDAAAAAAAQkXIGAAAAAAAgIuUMAAAAAABARMoZAAAAAACAiJQzAAAAAAAAESlnAAAAAAAAIlLOAAAAAAAARKScAQAAAAAAiEg5AwAAAAAAEJFyBgAAAAAAICLlDAAAAAAAQETKGQAAAAAAgIiUMwAAAAAAABEpZwAAAAAAACJSzgAAAAAAAESknAEAAAAAAIhIOQMAAAAAABCRcgYAAAAAACAi5QwAAAAAAEBEpTEXq+2WL1+efPDBB/nHrVu3TkpL/ecBvq28vDyZO3du/nHnzp2Txo0bJ8XAMRCoq8dAxz8glGMgUFcV4/EvxzEQqMljoCPOGnIH4z322KOmtwEUkDfffDPp0aNHUgwcA4G6egx0/AO+C8dAoK4qluNfjmMgUJPHQJc1AwAAAAAAiEg5s4bc6YsAdfW4UUzPBYijWI4bxfI8gLiK5dhRLM8DiKeYjhvF9FyAwjtuKGfW4LqSQF0+bhTTcwHiKJbjRrE8DyCuYjl2FMvzAOIppuNGMT0XoPCOG8oZAAAAAACAiJQzAAAAAAAAESlnAAAAAAAAIlLOAAAAAAAARKScAQAAAAAAiEg5AwAAAAAAEJFyBgAAAAAAICLlDAAAAAAAQETKGQAAAAAAgIiUMwAAAAAAABEpZwAAAAAAACJSzgAAAAAAAESknAEAAAAAAIhIOQMAAAAAABCRcgYAAAAAACAi5QwAAAAAAEBEyhkAAAAAAICIlDMAAAAAAAARKWcAAAAAAAAiUs4AAAAAAABEpJwBAAAAAACISDkDAAAAAAAQkXIGAAAAAAAgIuUMAAAAAABARMoZAAAAAACAiEpjLgYA/HfnnXdeaqZJkyZBs7p06ZKaGThwYJKlW265JTUzYcKEoFn33XdfBjsCAAAAqH2cOQMAAAAAABCRcgYAAAAAACAi5QwAAAAAAEBEyhkAAAAAAICIlDMAAAAAAAARKWcAAAAAAAAiUs4AAAAAAABEpJwBAAAAAACIqDTmYgBQVz388MNBuYEDByYxVVRUZDrvtNNOS8307t07aNa4ceNSM7NmzQqaBVAb7LTTTkG5KVOmpGbOPvvsoFk33HBDUA4oHs2aNUvNXH311Zl9bZfz9ttvp2YGDRoUNGvmzJlBOQAodM6cAQAAAAAAiEg5AwAAAAAAEJFyBgAAAAAAICLlDAAAAAAAQETKGQAAAAAAgIiUMwAAAAAAABEpZwAAAAAAACJSzgAAAAAAAESknAEAAAAAAIioNOZiAFCMHn744dTMwIEDk9imTJmSmnnuueeCZm2//fZBuf79+6dmdthhh6BZxxxzTGrmyiuvDJoFUBvstttuQbmKiorUzOzZszPYEVCM2rZtm5o55ZRTMjse5XTv3j01069fv6BZN910U1AOKB7dunVLzTz66KNBs9q3b5/BjopDnz59UjOTJ08OmvX3v/89gx3xn5w5AwAAAAAAEJFyBgAAAAAAICLlDAAAAAAAQETKGQAAAAAAgIiUMwAAAAAAABEpZwAAAAAAACJSzgAAAAAAAESknAEAAAAAAIioNOZiAFBIdt9996DcEUcckdmaH330UWrm0EMPDZo1b9681ExZWVnQrIYNGwblXn/99dTMrrvuGjSrZcuWQTmAQtG1a9eg3JIlS1Izjz32WAY7AgpJ69atg3L33HNPte8FIEsHHXRQaqZRo0ZR9lJM+vfvn5o56aSTgmYdddRRGeyI/+TMGQAAAAAAgIiUMwAAAAAAABEpZwAAAAAAACJSzgAAAAAAAESknAEAAAAAAIhIOQMAAAAAABCRcgYAAAAAACAi5QwAAAAAAEBEyhkAAAAAAICISmMuRu0xcODAoNwpp5wSlPv8889TM8uXLw+a9cADD6Rm/vGPfwTN+vjjj4NyAGvTtm3boFxJSUlq5qOPPgqaddBBB6Vm5syZk8R27rnnBuV22WWXzNZ8+umnM5sFUN06deqUmjnzzDODZt13330Z7AgoJL/4xS9SM4cffnjQrD322COpjfbbb7+gXL166a8jfv/994NmjR8/PigHVI/S0rAfPfft27fa91IXvf3226mZc845J2hWs2bNgnJLliwJyvEvzpwBAAAAAACISDkDAAAAAAAQkXIGAAAAAAAgIuUMAAAAAABARMoZAAAAAACAiJQzAAAAAAAAESlnAAAAAAAAIlLOAAAAAAAARKScAQAAAAAAiKg05mLUHiNGjAjKtW/fPonttNNOS80sXrw4aNZHH32UwY7qjtmzZ2f652fixIkbuCOoWU8++WRQbscdd8zsuPXVV18ltdFRRx0VlGvQoEG17wWgNtp5551TM82aNQua9fDDD2ewI6CQ/O///m9qpqKiIilkRx55ZGa5mTNnBs0aPHhwaubtt98OmgWsvx/+8IdBub322iuzn0Xxb5tttllqZpdddgma1bRp06DckiVLgnL8izNnAAAAAAAAIlLOAAAAAAAARKScAQAAAAAAiEg5AwAAAAAAEJFyBgAAAAAAICLlDAAAAAAAQETKGQAAAAAAgIiUMwAAAAAAABGVxlyM2uOUU04JynXp0iUoN3ny5NTM97///aBZ3bp1S83sv//+QbP23HPP1Mzf//73oFnbbLNNElt5eXlqZu7cuUGz2rZtm2Rl1qxZQbmJEydmtibUZjNnzkwK1ZAhQ4JyO+20U2ZrvvHGG5nmAGqD888/P7N/L3wNBcVj9OjRQbl69Qr7tbPz589PzZSVlQXNateuXWpmu+22C5r15ptvpmbq168fNAv4pk6dOqVmHnzwwaBZ06dPT81cccUVQbP4t8MOO6ymt0CKwv7XHwAAAAAAoMAoZwAAAAAAACJSzgAAAAAAAESknAEAAAAAAIhIOQMAAAAAABCRcgYAAAAAACAi5QwAAAAAAEBEyhkAAAAAAICIlDMAAAAAAAARlcZcjNpjzJgxmeZCPPvss5nN2myzzYJyXbt2Tc28/fbbQbN69OiRxLZ8+fLUzN/+9regWZMnT07NtGjRImjW9OnTg3JAzerXr19q5tJLLw2a1bBhw6Dcl19+mZr59a9/HTRr6dKlQTmA6tS+ffug3O67757Z121LliwJygE1q1evXqmZDh06BM2qqKjIJJO1W2+9NSj3/PPPp2a+/vrroFkHHHBAauaiiy5KsvLzn/88KHfLLbdktiYUg6FDh6ZmmjVrFjTr4IMPTs2UlZUFzaoLQn9+F/LvVE3828K/OXMGAAAAAACgmM+cOemkk/61cGlpctttt33nOaeffnr+rIKSkpLkj3/8Y4Y7BAAAAAAAKKJy5u67784XKvXr19+gcua+++6rutyJcgYAAAAAACgUNXJZs8rKylo1BwAAAAAAIBb3nAEAAAAAAIioYMuZFStW5N83bNiwprcCAAAAAABQ3OXMP/7xj2TlypX5e9dsvPHGNb0dAAAAAACA4i5nbrvttqrHO+ywQ43uBQAAAAAAYH2UJtVg3Lhx+bf/pqKiIrn00kuDZ65atSpZsGBB8tZbbyVvvvlm1cf33nvvDdorhSn3ZyHE2LFjM1tzzJgxSW00YMCAoNxmm22Wmvnggw+CZj388MNBOaBm7b777qmZrC8PGnJ8SPsaAaA26dWrV2az5s6dm9ksoPq0b98+KPfQQw+lZlq1apXENnPmzNTMyJEjg2b99re/DcotXbo0ibn/U089NWhW69atUzMjRowImtW4ceOg3I033piayV0NBmqrgQMHBuX69u2bmvn444+DZk2cODEox79cdNFFQbncz9/TvPTSS0GzFi5cGJSjFpQzuf+puX/Ac5cdW5fKysrgf+TXpV69esmJJ564QTMAAAAAAAAKvpypTrlSZ3Uxk3t1Q6dOnWp6SwAAAAAAALWjnFldpKzv59amtLQ02XjjjfOnF++1117JySefnOy2224Z7BIAAAAAAKDAy5lhw4bl39Ymd8ZL7nJn9evXT1asWFEdywMAAAAAANRa9Wpi0fU9awYAAAAAAKBYRL/nzPHHH59/nztzBgAAAAAAoK6JXs7cddddsZcEAAAAAACo25c1AwAAAAAAqKuinznzXe5PM378+OT9999PVqxYkbRv3z45+OCDk4022qimtwYAAAAAAFD7y5kZM2Ykb731Vv5x06ZNk379+q0z+9e//jU56qijkqlTp37j402aNEmuvPLK5Kyzzqr2/UJN2nzzzVMzN998c9CsevXST5S79NJLg2Z99dVXQTmgejz++ONBuT59+mS25r333huUGzp0aGZrAtQGnTt3zmzWiBEjMpsFVJ/S0rAflbRq1SqJady4cUG53M9R0sybNy+prWbOnJmayf1MKMS1116bmsn9bCrLY/ioUaNSM9OnTw+aBTVh0KBBQbmQvzuhP7Pi33InJqQ55phjgmatWrUqNXP55ZcHzVq5cmVQjlp+WbPLLrssOfroo/NvTz311Dpzs2bNSvbff/98MZM7e2a13OOlS5cmv/zlL4P/8AAAAAAAANTZcubpp5+uKltOOumkdebOOeecZOHChfnHJSUl+d+z+vet/vVvf/vb5L333ou0cwAAAAAAgAIrZ3KXNJs7d26+XGnZsmWyxx57rDP32GOP5XM5Xbp0yZ9lM2nSpOSWW27J328m97mKiopk+PDhMZ8CAAAAAABA4dxzZsqUKVWPu3Xrts7cQw89VHWWTO6eG+PHj0823njj/K933nnn/McGDBhQdSZO7jJnodcIBQAAAAAAqDNnzuTuI7Nahw4d1pl74YUX8u9zZ8fkLn22uphZ7Ygjjkh23HHH/ON//vOfybvvvlttewYAAAAAACjYcmbRokVVj5s3b77WzMqVK5PXX3+96tdHHnnkWnM9e/Zc6xk5AAAAAAAAtVnUcmbFihVVj+vXr7/WzDvvvJMsX748/3iTTTZJdt9997Xmtt5666rHCxcuzHyvAAAAAAAABV/ONGvWrOrxggUL1pp59dVXqy5pttdee61z1prlTu7SZgAAAAAAAIWgNOZibdq0qXo8adKktWaef/75qsf/rZxZ82yZpk2bZrZHqE3OOOOM1Ezr1q2DZq2rEF3T1KlTg2YB1adt27apmb333jtoVqNGjVIz8+bNC5p1+eWXB+XKysqCcgA1bc899wzKnXjiiUG5kPtgrr63JsB/mjhxYmomd0/eEKFf3xWyUaNGBeWOOeaY1EyPHj0y2BHUfuu6xcR3/RopxC233JLZrLri1FNPTc20atUqaNbkyZNTM2PHjg2aRRGcOdOlS5f8+8rKyuSVV1751hcMc+bMSV588cWqX/fq1Wuds2bOnLnW0gcAAAAAAKA2i1rOdOzYMdl2223zlyzLXYos96qPpUuX5j+Xu8/MKaeckpSXl1c1gPvss886Z+XuTbPaDjvsEGH3AAAAAAAABVbO5Jx22mn5M2dynn766WTrrbfOX55lq622Sp555pn8x3PlTa64WfO+Mmv6+OOPk1mzZuUfl5aWJp07d474DAAAAAAAAAqonDn33HPzlzdbXdDk7h3zxhtvfON+GFtuuWVywQUXrHPGyJEjq0qcXXfdNWnSpEmEnQMAAAAAABRgOdOwYcPk+eefz99PZnVBs+b79u3b58+o2XTTTdf6+3OZ22+/verXBx10UKSdAwAAAAAAbLjSpAZsvvnmydixY5OXX345+ctf/pJ88cUXyUYbbZTsscceyeGHH54vcNbl3XffTbbZZpv8W86gQYMi7hwAAAAAAKAAy5nV9t133/zb+ujWrVu+2AEAAAAAAChE0S9rBgAAAAAAUJcpZwAAAAAAAOrKZc2grtpnn32Ccv/v//2/zNbM3c8pzYcffpjZesB3M3LkyNRMy5YtM1vv/vvvD8pNnz49szUBaoPevXsH5Vq0aBGUe/bZZ1Mzy5cvD5oFFIZ69bJ7vesPfvCDzGbVBSUlJZn9P8ry/2POJZdckpo59thjM10TQjRq1Cgot9VWWwXlHnzwwQ3cEWuzww47ZDbLz/lqvxovZxYtWpQ88MAD+fvIvPfee8n8+fOTr7/+Ov+58vLyb+Vzn//ss8/yjxs2bJjsvPPO0fcMAAAAAABQcOVMZWVlMnz48OSaa65JFi9eXPWxtFdBfP7550nXrl3zn8+9uuGTTz5Jttlmm2j7BgAAAAAAKLh7zixbtizp06dPMmzYsPyZM7lSZnUxk3ZqaufOnfO/N5evqKhI7rvvvki7BgAAAAAAKNBy5phjjknGjBlT9etmzZolhx56aHLuuecG/f7jjz++6vFTTz1VLXsEAAAAAAAoinLmz3/+c/L4449XnSEzePDgZMaMGfmPXX311UE3Yuvfv3/SoEGD/NkzEydOTJYsWRJh5wAAAAAAAAVYzlx++eVVjwcMGJA8+OCDSatWrdZrxkYbbZR07Ngx/3jVqlXJRx99lPk+AQAAAAAACr6cmTVrVvLBBx/kHzdu3Di58cYbv/OsLl26VD3+29/+lsn+AAAAAAAAiqqcmTBhQv597pJm+++/f7L55pt/51mtW7euejx//vxM9gcAAAAAAFBU5cyXX35Z9bhTp04bNCt3abPV3HMGAAAAAAAoFKUxF1uzRGnatOkGzVq8eHHV42bNmm3QLIitb9++QbkGDRqkZsaMGbNeZ64BNePQQw8NynXr1i2zNV966aXUzLBhwzJbD6CQ7LrrrkG5ysrKoNwjjzyygTsCaouf/exnQbmKiopq3wtr179//6Dcbrvtltn/x9DcJZdcEpSD2Nb8Wep/89577633LSfWpUWLFkGzvvrqq6TYhV5BauDAgZmt+corr2Q2iyI4c6ZVq1aZ/aX79NNPqx63bNlyg2YBAAAAAAAUZTmz5ZZbrncLu65XK4wfP77q1zvttNMG7w0AAAAAAKDoypl99tknqV+/fv7SAK+//noyZ86c7zTnT3/6UzJv3ryqe8/svvvuGe8UAAAAAACgCMqZ5s2bJz179sw/Li8vT4YOHbreM7788svkvPPOyz8uKSlJ+vXrl9SrF/VpAAAAAAAAfGeZtxrbb799/q1Dhw5r/fyahczdd9+dXHnllcGzp02blhxwwAHJ559/XvWxCy64YAN3DAAAAAAAUMDlzKeffpp/mzFjxlo/f+CBByYDBgzIX9psdVnTq1ev5NFHH03mzp37jezKlSvzRcyTTz6ZnHzyyUnnzp2TyZMnV501c9pppyVdunTJ+ikAAAAAAABUm9KkBtx7773JzJkzk4kTJ+ZLlldeeSX/tqZcedO4ceNvfSyXz9lzzz2T6667Luq+AQAAAAAANlSN3KylSZMmydixY5PBgwfnC5fVZ9Gsfp8rYHJvqz+3+uOrM7kzb1544YWkQYMGNbF9AAAAAACAwjpzJqdZs2bJgw8+mBx11FHJiBEjkgkTJuQ/vmYRs6bcxzt27JhcfPHFyaBBgyLvFtavfExz8MEHB81asWJFambYsGFBs3KXCQSqR8uWLVMzF154YdCsLF948N5776VmysrKMlsPoLbYYostUjP77rtv0KypU6cG5R577LGgHFD79e/fv6a3UJRat24dlNtll10y+9o6S/95Kf518b03tdWyZcuCctOnTw/K5V48n+bpp58OmnXttdcmtVGnTp2Ccrn7r6dp37590Kx1/Wz8u6ioqMhsFkVWzqx22GGH5d8+/vjjZNy4cck777yTzJs3L1m4cGHStGnTpFWrVvl/mHP3qnF/GQAAAAAAoNDVeDmz2o477ph/O/nkk2t6KwAAAAAAAMV1zxkAAAAAAIC6SjkDAAAAAAAQkXIGAAAAAACg0O85U1JSkqxatSo54IADqmP8t9YaM2ZMta8DAAAAAABQa8uZnMrKymTcuHHVNb5qjVw5AwAAAAAAkNT1y5opTQAAAAAAACKfOQMAAAAAAECEciZXzJSWlibTpk2rjvFQqw0ZMiQ1s9tuuwXNevbZZ1Mzr732WtAsoPqce+65qZkePXpktt7jjz8elBs2bFhmawIUkhNOOCE1s/nmmwfNeuaZZzLYEQAXXXRRUO6MM85IYvr000+Dcscff3xQbtasWRu4I6hZod9Hhlw16ZBDDgma9eCDDya10bx58zI7SaFVq1ZJbHfffXf0NaklZ87ktGvXrjrHAwAAAAAAFJxqu+cMAAAAAAAA36acAQAAAAAAiEg5AwAAAAAAEJFyBgAAAAAAICLlDAAAAAAAQETKGQAAAAAAgEIvZ0pKSqpjLAAAAAAAQMErrY6hlZWV1TEWatQhhxwSlPvNb36Tmlm0aFHQrEsvvTQoB9Ssc845J+p6Z555ZlCurKys2vcCUBu1a9cus1kLFizIbBZAsRo9enRqpkOHDkltNGnSpKDcK6+8Uu17gdpgypQpQbmf/OQnqZmuXbsGzdpxxx2T2uiRRx7JbNY999wTlDvmmGMyW3PZsmWZzaJAypkZM2bk3zt7BgAAAAAAIEI5k+Wr1AAAAAAAAIpNtdxzBgAAAAAAgLVTzgAAAAAAAESknAEAAAAAAIhIOQMAAAAAABCRcgYAAAAAACAi5QwAAAAAAEBEyhkAAAAAAICISmMuBrVRy5Ytg3LXX399UK5+/fqpmdGjRwfNev3114NyQN3SokWLoNzKlSuT2ujrr7/ObP8NGjQImtW8efMkK5tuumlQ7pxzzkliWrVqVVDuggsuSM0sXbo0gx1BzenXr19ms5588snMZgGFoaSkJChXr152r3f98Y9/nNms2267LTWz5ZZbJlkK+W9RUVGR1Eb9+/ev6S1A0XrvvfcyzRWyTz75JPqanTp1Ss18+OGHUfbC2jlzBgAAAAAAICLlDAAAAAAAQETKGQAAAAAAgIiUMwAAAAAAABEpZwAAAAAAACJSzgAAAAAAAESknAEAAAAAAIhIOQMAAAAAABBRaczFILb69eunZp599tmgWdttt11Qbvr06amZ3/zmN0GzANbmr3/9a1LI/vznPwfl5syZk5pp06ZN0KzBgwcH5eqCf/zjH6mZ4cOHR9kLrK+ePXsG5bbYYotq3wtQvG655Zag3IgRIzJb86mnnkrNVFRUZLZelrNq85q33npr9DUB1qakpCTTXIgPP/wws1lUD2fOAAAAAAAARKScAQAAAAAAiEg5AwAAAAAAEJFyBgAAAAAAICLlDAAAAAAAQETKGQAAAAAAgIiUMwAAAAAAABEpZwAAAAAAACJSzgAAAAAAAERUGnMxiG2HHXZIzXTv3j3TNc8555zUzPTp0zNdE6hZo0ePTs0cdthhUfZSCAYNGpTURuXl5UG5ioqKzNYcNWpUUG7ixImZrfnyyy9nNgtiO+KII4Jy9evXT828++67QbPGjx8flAOKx6OPPhqUGzJkSGqmdevWGeyoOMydOzcoN3ny5NTMqaeeGjRrzpw5QTmA6lZZWZlpjuLgzBkAAAAAAICIlDMAAAAAAAARKWcAAAAAAAAiUs4AAAAAAABEpJwBAAAAAACISDkDAAAAAAAQkXIGAAAAAAAgIuUMAAAAAABARMoZAAAAAACAiEpjLgZZadeuXVDu+eefz2zNIUOGBOWeeuqpzNYECsORRx6Zmjn//PODZjVo0CCJqWPHjkG5wYMHJ7HdeeedqZlPP/00s/VGjhwZlJsyZUpmawL/1rRp09RM3759M1vvkUceCcqtWrUqszWBwjBz5syg3FFHHZWaOfzww4NmnX322UmxGz58eFDupptuqva9AMTWuHHjzGYtW7Yss1nULGfOAAAAAAAARKScAQAAAAAAiEg5AwAAAAAAEJFyBgAAAAAAICLlDAAAAAAAQETKGQAAAAAAgIiUMwAAAAAAABEpZwAAAAAAACIqjbkYZOXUU08Nym277baZrTlu3LigXGVlZWZrAsVjxIgRSSH7n//5n5reAlDkVq5cmZpZsGBB0KxRo0alZq677rqgWQDrMn78+EwyOc8//3xm3wf3798/k+Nkzm233RaUKykpSc1MmjQpaBZAMTrxxBODcgsXLkzNXHbZZRnsiNrAmTMAAAAAAAARKWcAAAAAAAAiUs4AAAAAAABEpJwBAAAAAACISDkDAAAAAAAQkXIGAAAAAAAgIuUMAAAAAABARMoZAAAAAACAiJQzAAAAAAAAEZXGXAxC9OzZMzVz1llnRdkLAABxrFy5MjWz9957R9kLQGzPPvtsJhkAaqe33norKHfttdemZsaOHZvBjqgNnDkDAAAAAAAQkXIGAAAAAAAgIuUMAAAAAABARMoZAAAAAACAiJQzAAAAAAAAESlnAAAAAAAAIlLOAAAAAAAARKScAQAAAAAAiKg05mIQYt99903NbLTRRpmtN3369KBcWVlZZmsCAAAAAHVD//79a3oL1ELOnAEAAAAAAIhIOQMAAAAAABCRcgYAAAAAACAi5QwAAAAAAEBEyhkAAAAAAICIlDMAAAAAAAARKWcAAAAAAAAiUs4AAAAAAABEpJwBAAAAAACIqDTmYhDb+++/n5o58MADg2Z99dVXGewIAAAAAIC6zpkzAAAAAAAAESlnAAAAAAAAIlLOAAAAAAAARKScAQAAAAAAiEg5AwAAAAAAEJFyBgAAAAAAICLlDAAAAAAAQETKGQAAAAAAgIhKYy4GIa688spMMgAAAAAAUBs5cwYAAAAAACAi5QwAAAAAAEBEyhkAAAAAAICIlDMAAAAAAAARKWcAAAAAAAAiUs4AAAAAAABEpJwBAAAAAACISDmzhvLy8preAlBgium4UUzPBYijWI4bxfI8gLiK5dhRLM8DiKeYjhvF9FyAwjtuKGfWMHfu3JreAlBgium4UUzPBYijWI4bxfI8gLiK5dhRLM8DiKeYjhvF9FyAwjtuKGcAAAAAAAAiKqmsrKyMuWBttnz58uSDDz7IP27dunVSWlpa01sCaunpi6tb8s6dOyeNGzdOioFjIFBXj4GOf0Aox0CgrirG41+OYyBQk8dA5QwAAAAAAEBELmsGAAAAAAAQkXIGAAAAAAAgIuUMAAAAAABARMoZAAAAAACAiJQzAAAAAAAAESlnAAAAAAAAIlLOAAAAAAAARKScAQAAAAAAiEg5AwAAAAAAEJFyBgAAAAAAICLlDAAAAAAAQETKGQAAAAAAgIiUMwAAAAAAABEpZwAAAAAAACJSzgAAAAAAAESknAEAAAAAAIhIOQMAAAAAAJDE8/8BeuxkZvxb/+gAAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": { "image/png": { "height": 321, "width": 819 } }, "output_type": "display_data" } ], "source": [ "samples_train = train_dataset.data[:5]\n", "samples_test = test_dataset.data[:5]\n", "plt.figure(figsize=(10, 4))\n", "for i in range(5):\n", " plt.subplot(2, 5, i+1)\n", " if i == 0:\n", " plt.ylabel('Train', fontsize=14)\n", " plt.imshow(samples_train[i], cmap='gray')\n", " plt.xticks([])\n", " plt.yticks([])\n", " plt.subplot(2, 5, i+6)\n", " if i == 0:\n", " plt.ylabel('Test', fontsize=14)\n", " plt.imshow(samples_test[i], cmap='gray')\n", " plt.xticks([])\n", " plt.yticks([])\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We need to build the dataloader: a function that cut the dataset in batches." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "batch_size = 100\n", "train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=100, shuffle=True)\n", "test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=100, shuffle=False)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "First batch: torch.Size([100, 1, 28, 28]) torch.Size([100])\n" ] } ], "source": [ "first_batch_images, first_batch_labels = next(iter(train_loader))\n", "print('First batch:', first_batch_images.shape, first_batch_labels.shape)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 2 Wrapper for training and Evaluation " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The goal of this section is to automatize the process of training and testing the model. \n", "\n", "To do so, we will define a function that will take as input the model, the loss function, the optimizer, the number of epochs, and the train and test loaders. The function will return the loss of the model during the training and testing phases. \n", "\n", "\n", "TODO: \n", "- Complete the AE_train function" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "def AE_eval(model, criterion):\n", " '''AE_eval\n", " Evaluate the model on the test set\n", " Input:\n", " - model: the trained model\n", " - criterion: the loss function\n", " Output:\n", " - test_loss: the average loss on the test set\n", " '''\n", " model.eval()\n", " test_loss = 0\n", " with torch.no_grad():\n", " for images, _ in test_loader:\n", " images = images\n", " outputs = model(images)\n", " test_loss += criterion(outputs, images).item()\n", " return test_loss/len(test_loader)\n", "\n", "def AE_train(model, criterion, optimizer, epochs=30, plot = True,verbose=False, freq=10):\n", " '''AE_train\n", " Train the model\n", " Input:\n", " - model: the model to train\n", " - criterion: the loss function\n", " - optimizer: the optimization algorithm\n", " - epochs: the number of epochs\n", " - plot: if True, plot the train and test losses\n", " - verbose: if True, print the losses\n", " - freq: frequency of printing the losses\n", " Output:\n", " - train_losses: the train losses\n", " - test_losses: the test losses\n", " '''\n", " if verbose:\n", " print('Start training...')\n", " train_losses=[]\n", " test_losses=[AE_eval(model, criterion)]\n", " for e in range(epochs):\n", " for batch_idx, (images, labels) in enumerate(train_loader):\n", " # Complete code here\n", " # Your code must include the following lines:\n", " # - Set the model to train mode\n", " # - Move the images to the device\n", " images = images.to(device)\n", " # - Reset the optimizer\n", " # - Forward pass\n", " # - Compute the loss\n", " # - Backward pass\n", " # - Update the weights\n", " # - Print the losses if verbose is True\n", " # - Evaluate the model at the end of each epoch\n", "\n", " if plot:\n", " plt.plot(train_losses, label='Train loss')\n", " plt.plot([e*batch_idx for e in range(e+2)], test_losses, label='Test loss', marker='o', linewidth=0.0)\n", " plt.legend()\n", " plt.xlabel('Batch')\n", " plt.ylabel('Loss')\n", " plt.show()\n", " return train_losses, test_losses\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 3 Linear Auto-encoder \n", "\n", "The basic idea is to have a linear transformation for both the encoder and decoder. \n", "For the output activation, we can use the sigmoid, since we know that the images are in greyscale, and the pixel values are between $0$ and $1$. \n", "\n", "The only hyperparameter we have is the dimension of the latent space: $h_{\\mathrm{dim}} < 784$.\n", "\n", "## 3.1 First model \n", "\n", "\n", "TODO: \n", "- Write a class for the encoder\n", "- Test it on some images as sanity-check using the *plot_result* function \n", "\n" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "### Write the class \n", "class AElin(nn.Module):\n", " '''AElin\n", " Autoencoder with linear layers\n", " Parameters:\n", " - hdim: the dimension of the hidden layer\n", " '''\n", " def __init__(self,hdim = 2):\n", " super().__init__()\n", " # Use the function nn.Sequential to define the encoder and decoder\n", " self.encoder = nn.Sequential(\n", " nn.Linear(28*28, hdim),\n", " )\n", " ... # Complete code here\n", " self.decoder = ... # Complete code here\n", " \n", " def encoder_function(self, x):\n", " x = x.view(x.size(0), -1)\n", " return self.encoder(x)\n", " \n", " def forward(self, x):\n", " x = x.reshape(x.size(0), -1)\n", " encoded = self.encoder(x)\n", " decoded = self.decoder(encoded)\n", " decoded = decoded.reshape(x.size(0), 1, 28, 28)\n", " return decoded\n", " \n", " " ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of parameters: 1570\n" ] }, { "ename": "TypeError", "evalue": "'ellipsis' object is not callable", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)", "Cell \u001b[0;32mIn[9], line 4\u001b[0m\n\u001b[1;32m 2\u001b[0m model \u001b[38;5;241m=\u001b[39m AElin()\n\u001b[1;32m 3\u001b[0m print_nparameters(model)\n\u001b[0;32m----> 4\u001b[0m \u001b[43mplot_result\u001b[49m\u001b[43m(\u001b[49m\u001b[43mmodel\u001b[49m\u001b[43m)\u001b[49m\n", "Cell \u001b[0;32mIn[2], line 9\u001b[0m, in \u001b[0;36mplot_result\u001b[0;34m(model)\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;21mplot_result\u001b[39m(model):\n\u001b[1;32m 2\u001b[0m \u001b[38;5;250m \u001b[39m\u001b[38;5;124;03m'''plot_result\u001b[39;00m\n\u001b[1;32m 3\u001b[0m \u001b[38;5;124;03m Plot the first 5 images of the test set and their reconstructions\u001b[39;00m\n\u001b[1;32m 4\u001b[0m \u001b[38;5;124;03m Input: \u001b[39;00m\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 7\u001b[0m \u001b[38;5;124;03m - None\u001b[39;00m\n\u001b[1;32m 8\u001b[0m \u001b[38;5;124;03m '''\u001b[39;00m\n\u001b[0;32m----> 9\u001b[0m first_outputs \u001b[38;5;241m=\u001b[39m \u001b[43mmodel\u001b[49m\u001b[43m(\u001b[49m\u001b[43mfirst_batch_images\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 10\u001b[0m plt\u001b[38;5;241m.\u001b[39mfigure(figsize\u001b[38;5;241m=\u001b[39m(\u001b[38;5;241m10\u001b[39m, \u001b[38;5;241m4\u001b[39m))\n\u001b[1;32m 11\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m i \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mrange\u001b[39m(\u001b[38;5;241m5\u001b[39m):\n", "File \u001b[0;32m~/DL2-IASO/venv/lib/python3.9/site-packages/torch/nn/modules/module.py:1773\u001b[0m, in \u001b[0;36mModule._wrapped_call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 1771\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_compiled_call_impl(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs) \u001b[38;5;66;03m# type: ignore[misc]\u001b[39;00m\n\u001b[1;32m 1772\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m-> 1773\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_call_impl\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n", "File \u001b[0;32m~/DL2-IASO/venv/lib/python3.9/site-packages/torch/nn/modules/module.py:1784\u001b[0m, in \u001b[0;36mModule._call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 1779\u001b[0m \u001b[38;5;66;03m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[1;32m 1780\u001b[0m \u001b[38;5;66;03m# this function, and just call forward.\u001b[39;00m\n\u001b[1;32m 1781\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m (\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_backward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_pre_hooks\n\u001b[1;32m 1782\u001b[0m \u001b[38;5;129;01mor\u001b[39;00m _global_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_backward_hooks\n\u001b[1;32m 1783\u001b[0m \u001b[38;5;129;01mor\u001b[39;00m _global_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[0;32m-> 1784\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mforward_call\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 1786\u001b[0m result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m 1787\u001b[0m called_always_called_hooks \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mset\u001b[39m()\n", "Cell \u001b[0;32mIn[8], line 24\u001b[0m, in \u001b[0;36mAElin.forward\u001b[0;34m(self, x)\u001b[0m\n\u001b[1;32m 22\u001b[0m x \u001b[38;5;241m=\u001b[39m x\u001b[38;5;241m.\u001b[39mreshape(x\u001b[38;5;241m.\u001b[39msize(\u001b[38;5;241m0\u001b[39m), \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m1\u001b[39m)\n\u001b[1;32m 23\u001b[0m encoded \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mencoder(x)\n\u001b[0;32m---> 24\u001b[0m decoded \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mdecoder\u001b[49m\u001b[43m(\u001b[49m\u001b[43mencoded\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 25\u001b[0m decoded \u001b[38;5;241m=\u001b[39m decoded\u001b[38;5;241m.\u001b[39mreshape(x\u001b[38;5;241m.\u001b[39msize(\u001b[38;5;241m0\u001b[39m), \u001b[38;5;241m1\u001b[39m, \u001b[38;5;241m28\u001b[39m, \u001b[38;5;241m28\u001b[39m)\n\u001b[1;32m 26\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m decoded\n", "\u001b[0;31mTypeError\u001b[0m: 'ellipsis' object is not callable" ] } ], "source": [ "# Define the model and test it \n", "model = AElin()\n", "print_nparameters(model)\n", "plot_result(model)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**TODO** \n", "- Train your model and plot the loss during training for 5 epochs.\n", "- Observe the result\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Model Initialization\n", "# model = AElin()\n", "print_nparameters(model)\n", "\n", "# Validation using MSE Loss function\n", "lossAE = nn.MSELoss()\n", "\n", "# Using an Adam Optimizer \n", "optimizer = optim.Adam(model.parameters(), lr=1e-4) \n", "\n", "AE_train(model, lossAE, optimizer, epochs=5, plot=True, verbose=True, freq=100)\n", "plot_result(model)\n", "plotLatentScatter(model.encoder_function, classes=[0,1,2,3,4,5,6,7,8,9])\n", "plotLatentScatter(model.encoder_function, classes=[0,1])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.2 Try different hyperpameters\n", "\n", "TODO: \n", "- Try different latent dimension: 2, 10, 20 50, 100\n", "- Try different learning rates: 1e-1, 1e-2, 1e-3, 1e-4, 1e-5\n", "\n", "For each try, plot the reconstructions result and the losses." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Hidden dimension \n", "for h in [2, 10, 20, 50, 100]:\n", "... # Complete code here" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "for lr in [1e-1, 1e-2, 1e-3, 1e-4, 1e-5]:\n", "\n", " ... # Complete code here " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 4 Use the AE as a denoising model\n", "\n", "We will see in this section how we can use the a AutoEncoder a denoising model. First, we define a function that will add noise to the images and plot the reconstruction." ] }, { "cell_type": "code", "execution_count": 56, "metadata": {}, "outputs": [], "source": [ "def plot_denoising(model):\n", " images = first_batch_images[:1]\n", " sigmas = [1e-2,2e-2, 5e-2, 1e-1,3e-1, 5e-1]\n", " plt.figure(figsize=(8, 4))\n", " for k, sigma in enumerate(sigmas):\n", "\n", " plt.subplot(3,len(sigmas), k+1)\n", " if k==0:\n", " plt.ylabel('Original', fontsize=14)\n", " plt.imshow(images.cpu().numpy().reshape(28, 28), cmap='gray')\n", " plt.xticks([])\n", " plt.yticks([])\n", "\n", " noise = torch.randn(1, 1, 28, 28) * sigma\n", " plt.subplot(3, len(sigmas), k+1+len(sigmas))\n", " if k == 0:\n", " plt.ylabel('Noisy', fontsize=14)\n", " plt.imshow((noise+images).cpu().numpy().reshape(28, 28), cmap='gray')\n", " plt.title(f'noise: {sigma:.0e}')\n", " plt.xticks([])\n", " plt.yticks([])\n", "\n", " plt.subplot(3, len(sigmas), k+1+2*len(sigmas))\n", " if k == 0:\n", " plt.ylabel('Reconstructed', fontsize=14)\n", " plt.imshow(model((images+noise)).cpu().detach().numpy().reshape(28, 28), cmap='gray')\n", " plt.xticks([]) \n", " plt.yticks([])\n", " plt.tight_layout()\n", " plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "TODO: \n", "- Compare the performance of three different models with $h_{\\mathrm{dim}}=10$, $h_{\\mathrm{dim}}=100$ and $h_{\\mathrm{dim}}=500$ " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "AE = ... # Complete code here \n", "optimizer = ... # Complete code here\n", "AE_train(AE, lossAE, optimizer, epochs=3, plot=False, verbose=False, freq=100);\n", "plot_denoising(AE)\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "AE = ... # Complete code here \n", "optimizer = ... # Complete code here\n", "AE_train(AE, lossAE, optimizer, epochs=3, plot=False, verbose=False, freq=100);\n", "plot_denoising(AE)\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "AE = ... # Complete code here \n", "optimizer = ... # Complete code here\n", "AE_train(AE, lossAE, optimizer, epochs=3, plot=False, verbose=False, freq=100)\n", "plot_denoising(AE)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "(BONUS) TODO: \n", "- Propose a new training procedure to train the model as a denoising model." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "\n", "def AE_train_denoising(model, criterion, optimizer, epochs=30, plot = True,verbose=False, freq=10):\n", " '''AE_train for denoising task\n", " Train the model\n", " Input:\n", " - model: the model to train\n", " - criterion: the loss function\n", " - optimizer: the optimization algorithm\n", " - epochs: the number of epochs\n", " - plot: if True, plot the train and test losses\n", " - verbose: if True, print the losses\n", " - freq: frequency of printing the losses\n", " Output:\n", " - train_losses: the train losses\n", " - test_losses: the test losses\n", " '''\n", "\n", " if verbose:\n", " print('Start training...')\n", " train_losses=[]\n", " test_losses=[AE_eval(model, criterion)]\n", " for e in range(epochs):\n", " for batch_idx, (images, labels) in enumerate(train_loader):\n", " ... # Complete code here \n", " if plot:\n", " plt.plot(train_losses, label='Train loss')\n", " plt.plot([e*batch_idx for e in range(e+2)], test_losses, label='Test loss', marker='o', linewidth=0.0)\n", " plt.legend()\n", " plt.xlabel('Batch')\n", " plt.ylabel('Loss')\n", " plt.show()\n", " return train_losses, test_losses\n", "\n", "\n", "AE = AElin(hdim=100)\n", "optimizer = optim.Adam(AE.parameters(), lr=1e-2)\n", "AE_train_denoising(AE, lossAE, optimizer, epochs=3, plot=True, verbose=False, freq=100)\n", "plot_result(AE)\n", "plot_denoising(AE)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 5 Anomaly detection\n", "\n", "Autoencoders can learn to represent the data in a latent representation space and to reconstruct the data from this compressed representation. One possible application of AE is Anomaly detection. The method is simple: \n", "- if you encode and decode an image similar to the training data, the reconstruction loss should be low, while \n", "- if you do the same with an image that differs from the training set, the loss should be higher. \n", "\n", "We will use a strong model:" ] }, { "cell_type": "code", "execution_count": 45, "metadata": {}, "outputs": [], "source": [ "AE = ... # Complete code here\n", "optimizer = ... # Complete code here\n", "AE_train( ... ); # Complete code here\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To test compare data dstribution we can compare:\n", "- the loss on the training data \n", "- the loss on the training with different anomalies\n", " - Transpositions\n", " - Artifacts\n", " - Mixing" ] }, { "cell_type": "code", "execution_count": 48, "metadata": {}, "outputs": [], "source": [ "def make_anomaly(input_data, mode='transpose'):\n", " '''make_anomaly\n", " Detect anomalies in the input data using the model\n", " Input:\n", " - input_data: the input data\n", " - mode: the mode for the anomaly detection among ['transpose', 'artifacts', 'mixing']\n", " Output:\n", " - anomalies: the pertubed images\n", " '''\n", " if mode == 'transpose':\n", " anomalies = input_data.clone()\n", " anomalies = anomalies.permute(0, 1, 3, 2)\n", "\n", " elif mode == 'artifacts':\n", " anomalies = input_data.clone()\n", " anomalies[:, :, 10:20, 10:20] = 1\n", " elif mode == 'mixing':\n", " anomalies = input_data.clone()\n", " anomalies[:, :, 10:15, 10:15] = input_data[:, :, 15:20, 15:20]\n", " anomalies[:, :, 15:20, 15:20] = input_data[:, :, 10:15, 10:15]\n", " anomalies[:, :, 10:15, 15:20] = input_data[:, :, 15:20, 10:15]\n", " anomalies[:, :, 15:20, 10:15] = input_data[:, :, 10:15, 15:20] \n", " return anomalies\n", "\n", "def plot_anomalies(model, mode='transpose'):\n", " anomalies = make_anomaly(first_batch_images, mode=mode)\n", " recons = model(first_batch_images)\n", " recons_a = model(anomalies)\n", "\n", " plt.figure(figsize=(8, 5))\n", " for i in range(5):\n", " plt.subplot(4, 5, i+1)\n", " image = first_batch_images[i].cpu().numpy().reshape(28, 28)\n", " if i == 0:\n", " plt.ylabel('Original', fontsize=14)\n", " plt.imshow(image, cmap='gray')\n", " plt.xticks([])\n", " plt.yticks([])\n", " plt.subplot(4, 5, i+6) \n", " recon = recons[i].cpu().detach().numpy().reshape(28, 28) \n", " if i == 0:\n", " plt.ylabel('Recons', fontsize=14)\n", " plt.title(f'MSE: {np.linalg.norm(image - recon):.2f}', fontsize=14)\n", " plt.imshow(recon, cmap='gray')\n", " plt.xticks([])\n", " plt.yticks([])\n", " plt.subplot(4, 5, i+11)\n", " anomaly = anomalies[i].detach().numpy().reshape(28, 28)\n", " if i == 0:\n", " plt.ylabel('Anomaly', fontsize=14)\n", " plt.imshow(anomaly, cmap='gray')\n", " plt.xticks([])\n", " plt.yticks([])\n", " plt.subplot(4, 5, i+16)\n", " recon = recons_a[i].cpu().detach().numpy().reshape(28, 28)\n", " if i == 0:\n", " plt.ylabel('Recons', fontsize=14)\n", " plt.title(f'MSE: {np.linalg.norm(anomaly - recon):.2f}', fontsize=14)\n", " plt.imshow(recon, cmap='gray')\n", " plt.xticks([])\n", " plt.yticks([]) \n", " plt.tight_layout()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot_anomalies(AE, mode='transpose')\n", "plot_anomalies(AE, mode='artifacts')\n", "plot_anomalies(AE, mode='mixing')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "TODO: \n", "- code a function that compute these two histograms and plot these\n", "- check if the histogram differs for the different type of anomalies" ] }, { "cell_type": "code", "execution_count": 52, "metadata": { "scrolled": true }, "outputs": [], "source": [ "\n", "def detect_anomalies(model,loader, mode='transpose'):\n", " '''detect_anomalies\n", " Plot the histogram of the MSE of the anomalies on the test set\n", " Input:\n", " - model: the trained model\n", " - mode: the mode for the anomaly detection among ['transpose', 'artifacts', 'mixing']\n", " Output:\n", " - None\n", " '''\n", " ... # Complete code here" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "detect_anomalies(AE, train_loader, mode='transpose')\n", "detect_anomalies(AE, train_loader, mode='artifacts')\n", "detect_anomalies(AE, train_loader, mode='mixing')\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 6 Full connected autoencoder\n", "\n", "We can increase the depth of the AE: \n", "- the encoder is composed of a first linear transformation, followed by a non-linearity (ReLU) and a second linear transformation. \n", "- the decoder performs the symmetric operations. \n", "\n", "As dimensions we can consider as a first choice : 784 # 256 ==> $h_{dim}=2$ ==> 256 ==> 784. \n", "\n", "TODO: \n", "- Write the class for such AE and test with the basic configuration (see above). \n", "- Train the model for $h_{dim}=2$ and $h_{dim}=100$ and compare the results." ] }, { "cell_type": "code", "execution_count": 54, "metadata": {}, "outputs": [], "source": [ "class SimpleAE(nn.Module):\n", " def __init__(self,hdim1=256,hdim2=2):\n", " super().__init__()\n", " ... # Complete code here\n", "\n", " def encoder_function(self, x):\n", " x = x.view(x.size(0), -1)\n", " return self.encoder(x)\n", "\n", " def forward(self, x):\n", " x = x.reshape(x.size(0), -1)\n", " encoded = self.encoder(x)\n", " decoded = self.decoder(encoded)\n", " decoded = decoded.reshape(x.size(0), 1, 28, 28)\n", " return decoded" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "SAE = ... # Complete code here\n", "optimizer = ... # Complete code here\n", "AE_train( ... ); # Complete code here\n", "plot_result(SAE)\n", "plotLatentScatter(SAE.encoder_function, classes=[0,1,2,3,4,5,6,7,8,9])\n", "plotLatentScatter(SAE.encoder_function, classes=[0,1,2,3,4,5,6,7,8,9])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "SAE = ... # Complete code here\n", "optimizer = ... # Complete code here\n", "_ = AE_train( ... ) # Complete code here\n", "plot_result(SAE)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "TODO:\n", "- Test the denoising and anomaly detection." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot_anomalies(SAE, mode='transpose')\n", "plot_anomalies(SAE, mode='artifacts')\n", "plot_anomalies(SAE, mode='mixing')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "detect_anomalies(SAE, train_loader, mode='transpose')\n", "detect_anomalies(SAE, train_loader, mode='artifacts')\n", "detect_anomalies(SAE, train_loader, mode='mixing')\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 7 Convolutional Autoencoder\n", "\n", "**TODO**:\n", "Implement a CNN Autoencoder with the following architecture:\n", "- Encoder:\n", " - Conv2d(1, 16, kernel_size=3, stride=2, padding=1) ==> ReLU ==> Conv2d(16, 32, kernel_size=3, stride=2, padding=1) ==> ReLU ==> Flatten ==> Linear(1568, $h_{dim}$)\n", "- Decoder:\n", " - Linear($h_{dim}$, 1568) ==> ReLu ==> Unflatten ==> ConvTranspose2d(32, 16, kernel_size=3, stride=2, padding=1, output_padding=1) ==> ReLU ==> ConvTranspose2d(16, 1, kernel_size=3, stride=2, padding=1, output_padding=1) ==> Sigmoid" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "class ConvAE(nn.Module):\n", " def __init__(self, hdim=2):\n", " super().__init__()\n", " ... # Complete code here\n", "\n", " def encoder_function(self, x):\n", " return self.encoder(x)\n", "\n", " def forward(self, x):\n", " encoded = self.encoder(x)\n", " decoded = self.decoder(encoded)\n", " return decoded" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "CAE = ConvAE()\n", "print_nparameters(CAE)\n", "plot_result(CAE)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "TODO:\n", "- Train the model for $h_{dim}=2$ and $h_{dim}=20$ and compare the results." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "CAE = ConvAE()\n", "optimizer = optim.Adam(CAE.parameters(), lr=1e-3)\n", "_ = AE_train(CAE, lossAE, optimizer, epochs=3, plot=True, verbose=True, freq=100)\n", "plot_result(CAE)\n", "plotLatentScatter(CAE.encoder_function, classes=[0,1,2,3,4,5,6,7,8,9])\n", "plotLatentScatter(CAE.encoder_function, classes=[0,1])\n", "plot_denoising(CAE)\n", "plot_anomalies(CAE, mode='transpose')\n", "plot_anomalies(CAE, mode='artifacts')\n", "plot_anomalies(CAE, mode='mixing')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "CAE = ConvAE(20)\n", "optimizer = optim.Adam(CAE.parameters(), lr=1e-3)\n", "_ = AE_train(CAE, lossAE, optimizer, epochs=3, plot=True, verbose=True, freq=100)\n", "plot_result(CAE)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot_denoising(CAE)\n", "plot_anomalies(CAE, mode='transpose')\n", "plot_anomalies(CAE, mode='artifacts')\n", "plot_anomalies(CAE, mode='mixing')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "detect_anomalies(CAE, train_loader, mode='transpose')\n", "detect_anomalies(CAE, train_loader, mode='artifacts')\n", "detect_anomalies(CAE, train_loader, mode='mixing')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "venv (3.9.6)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.6" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": true, "toc_window_display": false } }, "nbformat": 4, "nbformat_minor": 4 }