Supervised Deep Learning

Transcription

1 Supervised Deep Learning Joana Frontera-Pons Grigorios Tsagkatakis Dictionary Learning on Manifolds workshop Nice, September 2017

2 Supervised Learning Data Labels Model Prediction Spiral Exploiting prior knowledge Expert users Crowdsourcing Other instruments? Elliptical 2

3 State-of-the-art (before Deep Learning) Support Vector Machines Binary classification Kernels <-> non-linearities Random Forests Multi-class classification Markov Chains/Fields Temporal data 3

4 State-of-the-art (since 2015) Deep Learning (DL) Convolutional Neural Networks (CNN) <-> Images Recurrent Neural Networks (RNN) <-> NLP Long-Short Term memory (LSTM) <-> Audio 4

5 Convolutional Neural Networks (Convolution + Subsampling)+() + Fully Connected 5

6 height height Convolutional Layers 32x32x1 Image 28x28xK activation map 5x5x1 filter channels K filters 6

7 Convolutional Layers Characteristics Hierarchical features Location invariance Parameters Number of filters (32,64 ) Filter size (3x3, 5x5) Stride (1) Padding (2,4) Machine Learning and AI for Brain Simulations Andrew Ng Talk, UCLA,

8 Subsampling (pooling) Layers <-> downsampling Scale invariance Parameters Type Filter Size Stride 8

9 Activation Layer Introduction of non-linearity Brain: thresholding -> spike trains Tanh & Sigmoid 9

10 Activation Layer ReLU: x=max(0,x) Simplifies backprop Makes learning faster Avoids saturation issues ~ non-negativity constraint No saturated gradients Leaky ReLU: x=max(0.1x,x) 10

11 Fully Connected Layers Full connections to all activations in previous layer Typically at the end Can be replace by conv 11

12 Key Architectures LeNet [1998] AlexNet [2012] VGG GoogLeNet [2014] ResNet [2015] 12

13 LeNet 13

14 AlexNet Alex Krizhevsky, Ilya Sutskever and Geoff Hinton, ImageNet ILSVRC challenge in

15 K. Simonyan, A. Zisserman Very Deep Convolutional Networks for Large-Scale Image Recognition, arxiv technical report,

16 VGGnet D: VGG16 E: VGG19 All filters are 3x3 More layers smaller filters 16

17 Inception (GoogLeNet, 2014) Inception module Inception module with dimensionality reduction 17

18 Residuals 18

19 ResNet, 2015 He, Kaiming, et al. "Deep residual learning for image recognition." IEEE CVPR

20 Training protocols Fully Supervised Unsupervised pre-training + fine tuning Unsupervised pre-training + supervised layer 20

21 Success Stories Supervised learning application: Astronomy/Astrophysics Earth Observation Inverse problems Image super-resolution Image denoising 21

22 The Galaxy zoo challenge Online crowdsourcing project where users describe the morphology of galaxies based on color images 1 million galaxies imaged by the Sloan Digital Sky Survey (2007) 22

23 Dieleman, S., Kyle W. W., and Joni D.. "Rotation-invariant convolutional neural networks for galaxy morphology prediction." Monthly notices of the royal astronomical society,

24 Component 24

25 DL for galaxy morphology Recovery of galaxy parameters for HST images Simulation of 31K galaxies (24K training), H band PSF, CANDELS survey noise Tuccillo, D., Etienne Decencìère, and Santiago Velasco-Forero. "Deep learning for studies of galaxy morphology." Proceedings of the International Astronomical Union 12.S325 (2016):

26 DL for of galaxy morphology (con t) GALFIT CNN 26

27 CNN: Star-galaxy Classification Kim, Edward J., and Robert J. Brunner. "Star-galaxy classification using deep convolutional neural networks." Monthly Notices of the Royal Astronomical Society (2016): stw

28 Star Galaxy 28

29 Gravitational Lensing 29

30 CNN for lensing CNNs in Kilo Degree Survey colour-magnitude selected Luminous Red Galaxies, of which three are known lenses, the CNN retrieves 761 stronglens candidates and correctly classifies 2/3 of known lenses. Petrillo, C. E., et al. "Finding Strong Gravitational Lenses in the Kilo Degree Survey with Convolutional Neural Networks." Monthly Notices of the Royal Astronomical Society (2017). 30

31 DeepLens Training 20,000 LSST-like observations Testing for a rejection rate of non-lenses of 99%, a completeness of 90% can be achieved for lenses with Einstein radii larger than and S/N larger than 20 on individual g-band LSST exposures. Lanusse, Francois, et al. "CMU DeepLens: Deep Learning For Automatic Image-based Galaxy- Galaxy Strong Lens Finding." arxiv preprint arxiv: (2017). 31

32 Detecting strong lensing Strong galaxy-galaxy lensing systems CA-FR-HA Telescope Legacy Survey (CFHTLS) Ensemble of trained DL networks Search of 1.4 million early type galaxies selected from the survey catalog as potential deflectors, identified 2,465 candidates (117 previously known lens candidates, 29 confirmed lenses, 266 novel probable or potential lenses and 2097 false positives. Finding strong lenses in CFHTLS using convolutional neural networks 32

33 Other applications Classifying Radio Galaxies With Convolutional Neural Network Deep-hits: Rotation Invariant Convolutional Neural Network For Transient Detection 33

34 Beyond classification Inverse problems in Imaging Input: noisy/degraded image Output: clean/enhanced image Super-resolution 34

35 DL for super-resolution Patch Extraction and representation Non linear mapping Reconstruction C. Dong et al, Learning a deep convolutional network for image SR, ECCV

36 Relationship to CNNs SR creation stages Patch Extraction and representation Non linear mapping Reconstruction SR-CNN Applying a convolutional layer with n1 filters on the input image convolutional layers with a nonlinear activation Linear convolution on the n2 feature maps

37 Relationship to CNNs SR creation stages Patch Extraction and representation Non linear mapping Reconstruction SR-CNN Applying a convolutional layer with n1 filters on the input image convolutional layers with a nonlinear activation Linear convolution on the n2 feature maps

38 Relationship to CNNs SR creation stages Patch Extraction and representation Non linear mapping Reconstruction SR-CNN Applying a convolutional layer with n1 filters on the input image convolutional layers with a nonlinear activation Linear convolution on the n2 feature maps

39 Relationship to CNNs SR creation stages Patch Extraction and representation Non linear mapping Reconstruction SR-CNN Applying a convolutional layer with n1 filters on the input image convolutional layers with a nonlinear activation Linear convolution on the n2 feature maps

40 Impact of training examples 40

41 Residual for SR J. Kim et al, Accurate Image Super-Resolution Using Very Deep Convolutional Networks, arxiv: ,

42 Generative Adversarial Networks Generative model produces realistic new samples Discriminative model differentiate real vs synthetic samples Goodfellow, Ian, et al. "Generative adversarial nets." NIPS

43 GANs Key idea Training Use SGD-like algorithm on two minibatches : A minibatch of training examples A minibatch of generated samples Optional: run k steps of one player for every step of the other player. 43

44 Radford, Alec, Luke Metz, and Soumith Chintala. "Unsupervised representation learning with deep convolutional generative adversarial networks." arxiv preprint arxiv: (2015). 44

45 GANs for SR Ledig, Christian, et al. "Photo-realistic single image super-resolution using a generative adversarial network." arxiv preprint arxiv: (2016). 45

46 GANs for deconvolution 4,550 SDSS images of nearby galaxies at 0:01 < z < 0:02 Schawinski, Kevin, et al. "Generative Adversarial Networks recover features in astrophysical images of galaxies beyond the deconvolution limit." arxiv preprint arxiv: (2017). 46

47 Potential in learning Transfer learning /Domain adaptation Meta-learning / one-shot learning 47

48 Programming environments Keras (κέρας horn) Python neural networks library François Google Minimalist/Highly modular CPU/GPU execution TensorFlow Released by Google (Brain) 2016 Tensor modeling Computation graph 48

49 Example in Keras high-level neural networks library written in Python capable of running on top of either TensorFlow or Theano developed with a focus on enabling fast experimentation eras.pdf 49

50 Depedencies Python 2.7+ numpy: fundamental package for scientific computing with Python scipy: library used for scientific computing and technical computing Matplotlib (Optional, recommended for exploratory analysis) HDF5 and h5py (Optional, required if you use model saving/loading functions) Theano/Tensorflow 50

51 Sequential models Sequential Layers in Keras Dense: fully connected NN layer Activation: Applies an activation function Dropout: Applies Dropout to the input. Dropout consists in randomly setting a fraction p of input units to 0 at each update during training time, which helps prevent overfitting Convolutional Layers Pooling Layers 51

52 Example import numpy as np from keras.models import Sequential from keras.layers import Dense, Dropout, Activation, Flatten, Convolution2D, MaxPooling2D from keras.datasets import mnist # Load pre-shuffled MNIST train and test sets (X_train, y_train), (X_test, y_test) = mnist.load_data() 52

53 Example Creating Sequential Model use constructor: model = Sequential([ Dense(32, input dim=784), Activation( relu ), Dense(10), Activation( softmax ), ]), or add layers via the.add() method: model = Sequential() model.add(dense(32, input dim=784)) model.add(activation( relu )) 53

54 Comment The model needs to know what input shape it should expect first layer in a Sequential model (and only the first, because following layers can do automatic shape inference) needs to receive information about its input shape 54

55 Thank you 55