Deep Learning intro and hands-on tutorial

Transcription

1 Deep Learning intro and hands-on tutorial Π Ε. ώ Π ώ. Π ΠΘ 1 / 53

2 Deep Learning 2 / 53

3 Δ Έ ( ) ω π μ, ώπ ώ π ω (,...), π ώ π π π π ω ω ώπ ώ π, (biologically plausible) Δ π ώώ π ώ Ε π θ! 3 / 53

4 Ε π Δ 3 main ingredients! ( Δ ) Δ ώ (loss/objective function) 4 / 53

5 1943: Π ώ θ ώ ώ (Warren McCulloch and Walter Pitts) : Π π π θ π ώ ( neuroscience-oriented) 1958: Perceptron (Frank Rosenblatt) 1975: Backpropagation : ώ π π ώ π π π Ο ώ (π.. SVMs) θ ώ π π π π ώ Δ π π ώ Δ π ώ 5 / 53

6 2000-Σ μ : Deep Learning π state-of-the-art π π tasks Χ ώ π ώ π π π μ ώ (GPUs) π π π ώ Μ π μ ω θ ώ π (big data) Ν μ π (relu, Adam,...) 6 / 53

7 Ε ώ 7 / 53

8 8 / 53

9 9 / 53

10 Deep Learning A π ώ Π ( ώ ) π information retrieval routing algorithms... Data mining, analytics,... Έ π π valuable skills ώ 10 / 53

11 Deep Learning Startups Affectiva (emotion recognition from videos or images) Gridspace (Conversationally-aware software, speech recognition, speaker identification, critical moment) 11 / 53

12 Deep Learning Startups Ditto Labs (brand identification) Nervana (silicon to software optimized framework for deep learning) Finance, energy, online services, / 53

13 Deep Learning Startups Deep Genomics (precision medicine, understanding diseases, developing genetic therapies) Indico (text+image analytics for developing new products) 13 / 53

14 Deep Learning Startups Enlitic (medical image analysis) Deep Instinct ( cybersecurity, detects, predicts and prevents advanced persistent threats in real time) MarianaIQ (B2B account-based marketing) 14 / 53

15 Deep Learning Η π π ώ θ Perceptron: 15 / 53

16 Deep Learning 2-layer MLP: 16 / 53

17 Deep Learning 3-layer MLP: 17 / 53

18 Deep Learning LeNet: 18 / 53

19 Deep Learning GoogleLeNet: ResNet-200: 200 layers / 53

20 π Δ From scratch (C, Java,...) MATLAB Ε ώ π π π π π (Deep Learning Frameworks) Torch, Theano, Tensorflow,... Caffe, CNTK, Darkenet,... π π Python 20 / 53

21 ώ π π Framework? Η π π π ώ π π π π ώ, π Η ώ π Δ ώ ώ GPUs π deployment ώ ώ π.. CUDA Έ ώ π ώ Έ ώ ώ (π π ) 21 / 53

22 GPU Computing Ο GPUs π ώ hardware π ώ ώ π π π ώ π π (π.. π ώώ, convolutions,...) π ώ Δ π π Η nvidia ώ π GPU computing ώ ( ώ, π.. bitcoin mining) 22 / 53

23 GPU Computing Ο π ώώ ώ π θ GPU θ π θ π (cublas, curand, cudnn,... ) θ ώ π π wrapper θ π π π (π.. python) Ε (ResNet-200, forward+backward pass): Dual Xeon E v3: ms Titan X (Pascal): 297 ms 75 π GPU π π θ ώ, ώ ώ. 23 / 53

24 GPU Computing Δ ώ π, deployment! Ο π π ώ ώ ώ π GPU-accelerated hardware (π.x. self-driving cars) 24 / 53

25 DL Frameworks 25 / 53

26 Deep Learning Frameworks Π π! Caffe CTNK Tensorflow Darknet Theano (Py)Torch Keras, Blocks, Lasagne, 26 / 53

27 Caffe π π GPU-accelerated θ Deep Learning π ώ π θ ώ ω π μμ μ Η μ π ώ ώ Δ π C++ Python Π μ ω image analysis Χ π projects large-scale μπ μ Caffe2 (supported by Facebook) 27 / 53

28 Tensorflow Google supported! Η μ ώ Π distributed Yπ mobile Π ώ π APIs π (Python, C++, Java,...) Όώ : Π π π π π Π ώ 28 / 53

29 CTNK Microsoft Cognitive Toolkit (CNTK) Ε μ μ π ω μ ω Π π ώ deployment APIs C++, Python, C# 29 / 53

30 Darknet Lightweight minimal framework ώώ C CUDA π π ώ ώ Δ π ( ) π π minimal dependencies ( ) π ώ, θ, 30 / 53

31 Theano θ ώώ Python Π ώ π ώ θ ώ Ο ώ ώ (ώ θ ώ ) Theano ώ : π ώ ώ π π (C) π π π π π CUDA GPU π handler python Large compilation cost, fast execution 31 / 53

32 PyTorch Π ώ Torch Python π ώ θ π π μ ( ώπ ) Π ώ ( ώ ώπ ώ π θ π Caffe/Caffe2) π π π GPU Π ώ π ώ ώ π 32 / 53

33 θ /Wrappers Π π π π π GPUs π deep learning models π θ π π π π Keras (Tensorflow, Theano, CTNK) Lasagne (Theano) Blocks (Theano) 33 / 53

34 Pre-trained models Η π ώ ώ π π ώ θ ώ π ώ π ώ ώ π ώ workstations ώ 4 high end π π frameworks π ώ pretrained ώ π ώ π internet ώ π 34 / 53

35 Π Framework ώ? Π! π ώ π θώ? Δ θ ώ π ώ? Π θ ώ? Ε? Deployment target? Χ pretrained models? 35 / 53

36 Darknet 36 / 53

37 Darknet Ε π π ώ π state-of-the-art π ώ pretrained ώ Ε ( Linux) π ώ π θ ώ 37 / 53

38 ώ ώ π pretrained ώ ώ (yolo-tiny, yolo, coco-tiny, coco-tiny) Η COCO ώ ώ ώ π π π π ώ / 53

39 ώ./darknet coco test cfg/yolo-coco.cfg data/yolo-coco.weights image.jpg 39 / 53

40 ώ predictions.png 40 / 53

41 Deep Dream (nightmares)./darknet-gpu nightmare cfg/vgg-conv.cfg data/vgg-conv.weights image.jpg 5 41 / 53

42 Deep Dream (nightmares)./darknet-gpu nightmare cfg/vgg-conv.cfg data/vgg-conv.weights image.jpg 8 42 / 53

43 Deep Dream (nightmares)./darknet-gpu nightmare cfg/vgg-conv.cfg data/vgg-conv.weights image.jpg / 53

44 Deep Dream (nightmares)./darknet-gpu nightmare cfg/vgg-conv.cfg data/vgg-conv.weights image.jpg / 53

45 RNN text generation./darknet rnn generate cfg/rnn.cfg data/shakespeare.weights -srand 0 -seed Hello there! -len Hello there! There s a good f r i e n d. 2 COUNTESS. I am sure you can ; you have a d e v i l were made on 3 her beauty, and the very night than he two f i g u r e s 4 the worthy Achalle, and you s h a l l make such a messenger. 5 I have not s i n g l e d on the f i e l d ; 6 The name o f Caesar s sun, when you should speak, 7 Which when they would be reveng d on him in t h e i r death 8 But t h a t I have r e c e i v d and say they 9 Had s t r a n g e l y to his kingdom, and give her, 10 Upon the f i r s t here are both beams to love. 45 / 53

46 Keras 46 / 53

47 Keras θ Python π backend Tensorflow (default), theano CTNK Ε pip install keras Ε backend (edit Α/.keras/keras.json) 47 / 53

48 Keras Π π ώ well-known datasets (π.. MNIST) 1 from keras. d a t a s e t s import mnist datasets ώ pre-processing 1 ( x_train, y _ t r a i n ), ( x _ t e s t, y _ t e s t ) = mnist. load_data ( ) 2 x _ t r a i n = x _ t r a i n. reshape (60000, 784) 3 x _ t e s t = x _ t e s t. reshape (10000, 784) 4 x _ t r a i n = x _ t r a i n. astype ( f l o a t 3 2 ) / x _ t e s t = x _ t e s t. astype ( f l o a t 3 2 ) / keras π utilities ώ tasks (π.. ώ π labels binary ) 1 y _ t r a i n = keras. u t i l s. t o _ c a t e g o r i c a l ( y_train, 10) 2 y _ t e s t = keras. u t i l s. t o _ c a t e g o r i c a l ( y _ t e s t, 10) 48 / 53

49 Neural Networks in less than 10 lines Model Definition 1 model = S e q u e n t i a l ( ) 2 model. add ( Dense (512, a c t i v a t i o n = r e l u, input_shape =(784,) ) ) 3 model. add ( Dropout ( 0. 2 ) ) 4 model. add ( Dense (512, a c t i v a t i o n = r e l u ) ) 5 model. add ( Dropout ( 0. 2 ) ) 6 model. add ( Dense (10, a c t i v a t i o n = softmax ) ) 7 model. compile ( l o s s = c a t e g o r i c a l _ c r o s s e n t r o p y, optimizer=adam( ), metrics =[ accuracy ] ) Training 1 h i s t o r y = model. f i t ( x_train, y_train, b a t c h _ s i z e =128, epochs =10, verbose =2, v a l i d a t i o n _ d a t a =( x _ t e s t, y _ t e s t ) ) Testing 1 score = model. e v aluate ( x _ t e s t, y _ t e s t, verbose =0) 49 / 53

50 Neural Networks in less than 10 lines θ default θώ ώ θ! 1 model. compile ( l o s s = c a t e g o r i c a l _ c r o s s e n t r o p y, optimizer=adam( l r =0.01), m etrics =[ accuracy ] ) learning rate epoch 1 epoch 5 epoch % 96.91% 97.83% % 97.95% 98.02% % 96.04% 96.67% % % 10.09% GPU vs CPU i (4 cores, 32 bit): 19s per epoch Geforce 1060: 2s per epoch 50 / 53

51 CNN in less than 10 lines Model Definition 1 model = S e q u e n t i a l ( ) 2 model. add (Conv2D(32, k e r n e l _ s i z e =(3, 3), a c t i v a t i o n = r e l u, input_shape=input_shape ) ) 3 model. add (Conv2D(64, ( 3, 3), a c t i v a t i o n = r e l u ) ) 4 model. add ( MaxPooling2D ( p o o l _ s i z e =(2, 2) ) ) 5 model. add ( Dropout (0.25) ) 6 model. add ( F l a t t e n ( ) ) 7 model. add ( Dense (128, a c t i v a t i o n = r e l u ) ) 8 model. add ( Dropout ( 0. 5 ) ) 9 model. add ( Dense ( num_classes, a c t i v a t i o n = softmax ) ) 51 / 53

52 MLP vs CNN GPU vs CPU i (4 cores, 32 bit): 157s per epoch Geforce 1060: 14s per epoch ώ MLP CNN MLP: 98.02% (2s per epoch) CNN: 99.19% (14s per epoch) 10 π 52 / 53

53 Ε! 53 / 53