MIXED PRECISION TRAINING ON VOLTA GPUS
|
|
|
- Daniel Banks
- 5 years ago
- Views:
Transcription
1 MIXED PRECISION TRAINING ON VOLTA GPUS Boris Ginsburg, Paulius Micikevicius, Oleksii Kuchaiev, Sergei Nikolaev bginsburg, pauliusm, okuchaiev, 10/17/2017
2 ACKNOWLEDGMENTS Michael Houston, Hao Wu, Ahmad Kiswani, Amir Gholaminejad, Ujval Kapasi, Jonah Alben, Alex Fit-Florea, Slawomir Kierat cudnn team Work is based on NVIDIA branch of caffe (caffe-0.16) 2
3 AGENDA 1. Mixed precision training with Volta TensorOps 2. Training without master-copy weights 3. NVCAFFE support for FLOAT16 3
4 TERMINOLOGY Training type Data type Matrix-Multiply Accumulator Weight update FP32 FP32 FP32 FP32 GPU Pascal FP16 FP16 FP16 FP16/FP32 Pascal(GP-100) Mixed precision FP16 FP32 FP16/FP32 Volta Master weights copy of weights used for update (SGD step). Volta: Mixed precision training with FP32 master weight storage. 4 4
5 HALF-PRECISION FLOAT (FLOAT16) float16 sign exponent (5 bit) fraction (10 bit) float sign exponent (8 bit) fraction (23 bit) FLOAT16 has wide range (2 40 ) but it is not as wide as FP32! Normal range: [ , ] Sub-normal range: [ , ] FP FLOAT 32 5
6 VOLTA: TENSOR CORE 4X4 MATRIX-MULTIPLY ACC 6
7 VOLTA TENSOR OPERATION FP16 storage/input Full precision product Sum with FP32 accumulator Convert to FP32 result more products F16 F16 + F32 F16 F32 Also supports FP16 accumulator mode for inferencing 7
8 TRAINING FLOW Yk Loss E de/dy k Yk Loss E Y k = W k *Y k-1 de/dy k-1 =de/dy k *W k de/dw k =de/dy k *Y k-1 de/dw k W k = W k -λ*de/dw k W k Y k = W k *Y k-1 Y 2 de/dy k-1 Y 2 Y 2 = W 2 *Y 1 de/dy 1 =de/dy 2 *W 2 de/dw 2 =de/dy 2 *Y 1 de/dw 2 W 2 = W 2 -λ*de/dw 2 W 2 Y 2 = W 2 *Y 1 Y 1 de/dy 1 Y 1 Y 1 = W 1 *X de/dx=de/dy 1 *W 1 de/dw 1 =de/dy 1 *X de/dw 1 W 1 = W 1 -λ*de/dw 1 W 1 Y 1 = W 1 *X X X FORWARD PASS BACKPROP WEIGHT UPDATE FORWARD PASS 8
9 TRAINING FLOW Yk Loss E de/dy k Yk Loss E Y k = W k *Y k-1 de/dy k-1 =de/dy k *W k de/dw k =de/dy k *Y k-1 de/dw k W k = W k -λ*de/dw k W k Y k = W k *Y k-1 Y 2 de/dy k-1 Y 2 Y 2 = W 2 *Y 1 de/dy 1 =de/dy 2 *W 2 de/dw 2 =de/dy 2 *Y 1 de/dw 2 W 2 = W 2 -λ*de/dw 2 W 2 Y 2 = W 2 *Y 1 Y 1 de/dy 1 Y 1 Y 1 = W 1 *X de/dx=de/dy 1 *W 1 de/dw 1 =de/dy 1 *X de/dw 1 W 1 = W 1 -λ*de/dw 1 W 1 Y 1 = W 1 *X X X FORWARD PASS BACKPROP WEIGHT UPDATE FORWARD PASS 9
10 TRAINING FLOW Yk Loss E de/dy k Yk Loss E Y k = W k *Y k-1 de/dy k-1 =de/dy k *W k de/dw k =de/dy k *Y k-1 de/dw k W k = W k -λ*de/dw k W k Y k = W k *Y k-1 Y 2 de/dy k-1 Y 2 Y 2 = W 2 *Y 1 de/dy 1 =de/dy 2 *W 2 de/dw 2 =de/dy 2 *Y 1 de/dw 2 W 2 = W 2 -λ*de/dw 2 W 2 Y 2 = W 2 *Y 1 Y 1 de/dy 1 Y 1 Y 1 = W 1 *X de/dx=de/dy 1 *W 1 de/dw 1 =de/dy 1 *X de/dw 1 W 1 = W 1 -λ*de/dw 1 W 1 Y 1 = W 1 *X X X FORWARD PASS BACKPROP WEIGHT UPDATE FORWARD PASS 10
11 TRAINING FLOW Yk Loss E de/dy k Yk Loss E Y k = W k *Y k-1 de/dy k-1 =de/dy k *W k de/dw k =de/dy k *Y k-1 de/dw k W k = W k -λ*de/dw k W k Y k = W k *Y k-1 Y 2 de/dy k-1 Y 2 Y 2 = W 2 *Y 1 de/dy 1 =de/dy 2 *W 2 de/dw 2 =de/dy 2 *Y 1 de/dw 2 W 2 = W 2 -λ*de/dw 2 W 2 Y 2 = W 2 *Y 1 Y 1 de/dy 1 Y 1 Y 1 = W 1 *X de/dx=de/dy 1 *W 1 de/dw 1 =de/dy 1 *X de/dw 1 W 1 = W 1 -λ*de/dw 1 W 1 Y 1 = W 1 *X X X FORWARD PASS BACKPROP WEIGHT UPDATE FORWARD PASS 11
12 VOLTA TRAINING METHOD W (F16) W Y k F16 F16 FWD F16 Y k+1 de/dy k F16 BWD-A F16 F16 W de/dy k+1 de/dw F16 BWD-W F16 F16 Y k de/dy k+1 F16 Master-W (F32) F32 Weight Update F32 Updated Master-W 12
13 SOME NETS WORKED OUT OF THE BOX TensorOp training matched the results of F32 training Same solver, hyper-parameters, and training schedule as F32 Image classification (ILSVRC12): GoogleNet, VGG-D, Inception v3, ResNet-50 Solver: SGD with momentum Language modeling and machine translation NMT Solver: ADAM 13 13
14 14 GOOGLENET 14
15 15 INCEPTION V1 15
16 16 RESNET-50 16
17 MIXED PRECISION TRAINING Networks that don t work out-of-box : Image classification: Alexnet, CaffeNet Detection nets: RNNs: Multibox SSD with VGG-D backbone: was not learning Faster R-CNN with VGG-D backbone: low map 68.5% vs 69.1% with F32 Seq2seq with attention: lagged behind F32 in perplexity biglstm: diverged after some training 17 17
18 MULTI-BOX SSD: ACTIVATION GRADIENT HISTOGRAM 18 18
19 MULTI-BOX SSD: ACTIVATION GRADIENT HISTOGRAM Become denormals in F16 Become 0 in F
20 MULTIBOX SSD: ACTIVATION GRADIENT HISTOGRAM Become denormals in F16 Become 0 in F16 Unused Overall FP16 range 20 20
21 MIXED PRECISION TRAINING: GLOBAL LOSS SCALING The problem is small ( vanishing ) activation gradients Solution: 1. scale the loss value by K to shift gradients to FP16 range 2. rescale gradients back by 1/K before weight update 3. New hyper-parameter: loss scale 21
22 SCALING LOSS AND GRADIENTS F32 training Loss scaled by 256 => gradients get scaled by 256 Clippy training, loss scaled by 256 Benefits: shift activation gradients into working range Most weight gradients become normalized values in F
23 ALEXNET : COMPARISON OF RESULTS Mode Top1, % Top5, % Fp FP16 training, no scaling FP16 training, scale = Mixed precision training, scale = Nvcaffe-0.16, DGX1, SGD with momentum, 100 epochs, batch=1024, no augmentation, 1 crop, 1 model 23
24 ALEXNET: TRAINING WITH SCALING FP16 training matches other training curves (TensorOp and FP32) 24
25 OBJECTS DETECTION Multibox-SSD map: F32: 76.9% TensorOp: 77.1%, loss scaled by 256; Without scaling: doesn t learn Faster-RCNN map: F32: 69.1% TensorOp: 69.7%, loss scaled by 256, without scaling: 68.5% 25 25
26 SEQ2SEQ TRAINING Neural machine translation: Ge-En NMT_ONE model: 1. 2-layer bi-directional encoder (512 LSTM) 2. Normalized Bahdanau attention 3. 4-layer decoder (512 LSTM) 26 26
27 SEQ2SEQ TRAINING Scaling recovered acuracy 27
28 SEQ2SEQ TRAINING NMT_ONE scaling factor: 32,768 How to reduce scaling factor? 28
29 OPEN SEQ2SEQ LOSS Logits shape: [batch_size, time, dimension] Targets shape: [batch_size, time] Loss avg = Avg batch (Avg timesteps crossentropy logits, targets ) Loss sum = Avg batch (SUM timesteps (crossentropy logits, targets )) More numerically-friendly loss function LARS algorithm allows smaller scale factors 29
30 OpenSeq2Seq training: Eval BLEU score SEQ2SEQ TRAINING NMT_ONE seq2seq training 200 NMT_ONE FP32 NMT_ONE HMMA S OpenSeq2Seq training: training loss With sum loss and LARS scaling factor: NMT_ONE FP32 NMT_ONE HMMA S512 30
31 EVAL BLEU SEQ2SEQ TRAINING GNMT-like 8-layer bi-directional encoder (1024 LSTM) GNMT-style normalized Bahdanau attention 8-layer decoder (1024 LSTM) residual connections Training loss GNMT-like FFMA GNMT-like HMMA S With sum loss and LARS scaling factor: GNMT-like FFMA GNMT-like HMMA S1024 [1] Wu, Yonghui, et al. "Google's neural machine translation system: Bridging the gap between human and machine translation." arxiv preprint arxiv: (2016). 31
32 LANGUAGE MODEL 1 Billion Word Language Benchmark BigLSTM: 2x8192 LSTM, 1024 Projection 800K word vocabulary Adagrad solver Based on Exploring the Limits of Language Modeling,
33 33 BIGLSTM: 2X8192 LSTM, 1024 PROJECTION 33
34 34 GUIDELINES FOR TRAINING WITH MIXED PRECISION / TENSOROPS 34
35 VOLTA MIXED PRECISION TRAINING Mixed Precision Training: FP16 storage and TensorOps for fwd/bwd pass: weights, activations, gradients Batch Normalization: FP16 data, FP32 math FP32 master-copy of weights for weights update WARNING Gradient may become too small for FP16 range Solved with new hyper parameter loss scaling 35 35
36 OBSERVATIONS ON GRADIENT VALUES FP16 range: 2 40 including denorms Gradient range is biased low vs standard FP16 range Max magnitude we ve seen was O(2 3 ) shift values without overflowing Maximum magnitudes: weight-grad >> activation-grad * for all the nets we ve looked at 36 36
37 37 MIXED PRECISION TRAINING WITHOUT MASTER WEIGHT COPY 37
38 ALEXNET: FP16 MASTER WEIGHT STORAGE Can we avoid two weights copies? Can FP16 be used for weight update? Vanilla SGD weights update: W(t+1) = W(t) - λ*δw(t) If we use FP16 for ΔW, the product λ* ΔW(t) can become too small: Initially gradients ΔW(t) are very small. They are multiplied by learning rate λ which is < 1, so λ*δw(t)can go into subnormal float16 range Later gradients becomes larger, but λ becomes smaller, so λ*δw(t) becomes even smaller. 38
39 ALEXNET: FP16 MASTER WEIGHT STORAGE There are a number of solutions for this vanishing update problem. Option1 FP32 master copy of weights float W 32 master copy for updates, fp16 W 16 weights used for forward-backward pass: Compute ΔW 16 (t) using forward-backward pass Convert gradients to float: Update weights in float: ΔW 32 (t) =half2float(δw 16 (t)) W 32 (t+1)=w 32 (t) - λ*δw 32 (t) Make float16 copy of weights: W 16 (t+1)=float2half(w 32 (t+1)) Do forward-backward with W So W 32 will accumulate small weights updates. 39
40 ALEXNET: FP16 MASTER WEIGHT STORAGE Consider SGD with momentum: 1. Compute momentum H: H(t+1)= m*h(t) - λ*δw(t) 2. Update weights with H: W(t+1)= W(t) + H(t+1) λ is small, so λ*δw(t) can be very small and it can vanish if we compute momentum in float16. Can we fix this? Denote D(t)=ΔW(t). Assume for simplicity that λ = const. Then H(t+1)= m*h(t)-λ*d(t)= m*(h(t-1)-λ*d(t-1)) - λ*d(t)= -λ*[d(t) + m*d(t-1) + m 2 *D(t-2) + m k *D(t-k)+ ] Moment works as average of gradients! 40
41 ALEXNET: FP16 MASTER WEIGHT STORAGE Let s modify the original momentum schema: 1. Compute momentum H: H(t+1)= m*h(t) - λ*δw(t) 2. Update weights with H: W(t+1)= W(t) + H(t+1) ΔW 41
42 ALEXNET: FP16 MASTER WEIGHT STORAGE Let s modify the original momentum schema: 1. Compute momentum G: G(t+1)= m*g(t) + -λ ΔW(t) 2. Update weights with G: W(t+1)= W(t) λ*g(t+1) Now G will accumulate average of ΔW(t) which don t vanish! Weights update in float16: Compute Δw16(t) using forward-backward pass Compute momentum: Update in float math: Convert result to float16: G16(t+1) = m* G16(t) + Δw16(t) W=half2float(W16(t))- λ*half2float(g16(t+1)) W16(t+1)=float2half(W) Do forward-backward with W
43 ALEXNET: FP16 MASTER WEIGHT STORAGE With this fix we can have only one copy of weights in float16: 43
44 ALEXNET : COMPARISON OF RESULTS Mode Top1, % Top5, % Fp FP16 training, no scaling FP16 training, scale = Mixed precision training, scale = Mixed precision training, scale =1000 FP16 master weights
45 INCEPTION-V3 RESULTS Scale loss function by 100x Mode Top1, % Top5, % Fp Mixed precision training FP16 training FP16 training, loss scale = FP16 training, loss scale = 100, FP16 master weights
46 INCEPTION-V3 RESULTS 46
47 RESNET-50 RESULTS No scale Mode Top1, % Top5, % Fp FP16 training, FP32 master weights FP16 training, FP16 master weight Mixed, FP16 master weight Nvcaffe-0.16, DGX-1, SGD with momentum, 100 epochs, batch=512, min augmentation, 1 crop, 1 model 47
48 RESNET-50 RESULTS FP16 training is ok FP16 storage has a small dip at the end (noise?) 48
49 Volta Mixed Precision Training 1. Mixed precision with FP32 master weight storage: Good results on with a variety of networks Applying a global scaling to the loss input is needed for some networks Wide range of loss scaling values work well float master copy of weights 2. FP16 master weight storage worked for convnets after refactoring the solver 49
50 50 TRAINING WITH MIXED PRECISION IN NVCAFFE
51 NVIDIA/CAFFE-0.16 Full float16 support Mixed precision: Different data types for Forward and Backward Different math type Solver_type (for weight update in float16) Automatic type conversion 51
52 name: "AlexNet_fp16" default_forward_type: FLOAT16 default_backward_type: FLOAT16 NVIDIA/CAFFE-0.16 # default_forward_math: FLOAT # GP100 only # default_backward_math: FLOAT # GP100 only global_grad_scale: layer { forward_math: FLOAT16 backward_math: FLOAT } solver_data_type: FLOAT
53
54 NVIDIA/CAFFE-0.16 DATA AND MATH TYPES default_forward_type: FLOAT16 default_backward_type: FLOAT16 template<typename Ftype, typename Btype> class Layer : public LayerBase { default_forward_math: FLOAT forward_math_ = this->layer_param().forward_math(); setconvolutiondesc(forward_math_, fwd_conv_descs_[i], pad_h, pad_w, stride_h, stride_w); 54
55 NVIDIA/CAFFE-0.16 INTERNALS enum Type { DOUBLE = 0, FLOAT = 1, FLOAT16 = 2, class Blob { mutable shared_ptr<tensor> data_tensor_; mutable shared_ptr<tensor> diff_tensor_; class Tensor { Type type_; shared_ptr<vector<shared_ptr<syncedmemory>>> synced_arrays_; template<typename Dtype> class TBlob : public Blob { 55
56 NVIDIA/CAFFE-0.16 FUSED KERNELS template<typename Gtype, typename Wtype> global void SGDRegUpdateAllAndClear(int N, Gtype* g, Wtype* w, Wtype* h, float momentum, float local_rate, float local_decay, bool reg_l2, bool clear_grads) { CUDA_KERNEL_LOOP(i, N) { Wtype reg = reg_l2? w[i] : Wtype((Wtype(0) < w[i]) - (w[i] < Wtype(0))); Wtype gr = Wtype(g[i]) + reg * local_decay; gr = h[i] = momentum * h[i] + local_rate * gr; w[i] -= gr; g[i] = clear_grads? Gtype(0) : Gtype(gr); } } template<> global void SGDRegUpdateAllAndClear< half, float>(int N, half* g,float* w, float* h, float momentum, float l_rate, float l_decay, bool reg_l2, bool clear_grads) { half hz; hz.x = 0; CUDA_KERNEL_LOOP(i, N) { float reg = reg_l2? w[i] : (0.F < w[i]) - (w[i] < 0.F); float gr = half2float(g[i]) + reg * l_decay; gr = h[i] = momentum * h[i] + l_rate * gr; w[i] -= gr; g[i] = clear_grads? hz : float2half_clip(h[i]); } } 56
Lecture 17: Neural Networks and Deep Learning
UVA CS 6316 / CS 4501-004 Machine Learning Fall 2016 Lecture 17: Neural Networks and Deep Learning Jack Lanchantin Dr. Yanjun Qi 1 Neurons 1-Layer Neural Network Multi-layer Neural Network Loss Functions
Google s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Google s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation Y. Wu, M. Schuster, Z. Chen, Q.V. Le, M. Norouzi, et al. Google arxiv:1609.08144v2 Reviewed by : Bill
CSC321 Lecture 16: ResNets and Attention
CSC321 Lecture 16: ResNets and Attention Roger Grosse Roger Grosse CSC321 Lecture 16: ResNets and Attention 1 / 24 Overview Two topics for today: Topic 1: Deep Residual Networks (ResNets) This is the state-of-the
Recurrent Neural Networks (Part - 2) Sumit Chopra Facebook
Recurrent Neural Networks (Part - 2) Sumit Chopra Facebook Recap Standard RNNs Training: Backpropagation Through Time (BPTT) Application to sequence modeling Language modeling Applications: Automatic speech
Don t Decay the Learning Rate, Increase the Batch Size. Samuel L. Smith, Pieter-Jan Kindermans, Quoc V. Le December 9 th 2017
Don t Decay the Learning Rate, Increase the Batch Size Samuel L. Smith, Pieter-Jan Kindermans, Quoc V. Le December 9 th 2017 slsmith@ Google Brain Three related questions: What properties control generalization?
Convolutional Neural Networks II. Slides from Dr. Vlad Morariu
Convolutional Neural Networks II Slides from Dr. Vlad Morariu 1 Optimization Example of optimization progress while training a neural network. (Loss over mini-batches goes down over time.) 2 Learning rate
LARGE BATCH TRAINING OF CONVOLUTIONAL NET-
LARGE BATCH TRAINING OF CONVOLUTIONAL NET- WORKS WITH LAYER-WISE ADAPTIVE RATE SCALING Anonymous authors Paper under double-blind review ABSTRACT A common way to speed up training of deep convolutional
Introduction to Convolutional Neural Networks (CNNs)
Introduction to Convolutional Neural Networks (CNNs) nojunk@snu.ac.kr http://mipal.snu.ac.kr Department of Transdisciplinary Studies Seoul National University, Korea Jan. 2016 Many slides are from Fei-Fei
Recurrent Neural Networks with Flexible Gates using Kernel Activation Functions
2018 IEEE International Workshop on Machine Learning for Signal Processing (MLSP 18) Recurrent Neural Networks with Flexible Gates using Kernel Activation Functions Authors: S. Scardapane, S. Van Vaerenbergh,
arxiv: v3 [cs.cl] 24 Feb 2018
![arxiv: v3 [cs.cl] 24 Feb 2018](/thumbs/82/86952145.jpg "arxiv: v3 [cs.cl] 24 Feb 2018")
FACTORIZATION TRICKS FOR LSTM NETWORKS Oleksii Kuchaiev NVIDIA okuchaiev@nvidia.com Boris Ginsburg NVIDIA bginsburg@nvidia.com ABSTRACT arxiv:1703.10722v3 [cs.cl] 24 Feb 2018 We present two simple ways
Convolutional Neural Network Architecture
Convolutional Neural Network Architecture Zhisheng Zhong Feburary 2nd, 2018 Zhisheng Zhong Convolutional Neural Network Architecture Feburary 2nd, 2018 1 / 55 Outline 1 Introduction of Convolution Motivation
Day 3 Lecture 3. Optimizing deep networks
Day 3 Lecture 3 Optimizing deep networks Convex optimization A function is convex if for all α [0,1]: f(x) Tangent line Examples Quadratics 2-norms Properties Local minimum is global minimum x Gradient
Deep Learning Sequence to Sequence models: Attention Models. 17 March 2018
Deep Learning Sequence to Sequence models: Attention Models 17 March 2018 1 Sequence-to-sequence modelling Problem: E.g. A sequence X 1 X N goes in A different sequence Y 1 Y M comes out Speech recognition:
Deep Learning & Artificial Intelligence WS 2018/2019
Deep Learning & Artificial Intelligence WS 2018/2019 Linear Regression Model Model Error Function: Squared Error Has no special meaning except it makes gradients look nicer Prediction Ground truth / target
Introduction to RNNs!
Introduction to RNNs Arun Mallya Best viewed with Computer Modern fonts installed Outline Why Recurrent Neural Networks (RNNs)? The Vanilla RNN unit The RNN forward pass Backpropagation refresher The RNN
with Local Dependencies
CS11-747 Neural Networks for NLP Structured Prediction with Local Dependencies Xuezhe Ma (Max) Site https://phontron.com/class/nn4nlp2017/ An Example Structured Prediction Problem: Sequence Labeling Sequence
CS 229 Project Final Report: Reinforcement Learning for Neural Network Architecture Category : Theory & Reinforcement Learning
CS 229 Project Final Report: Reinforcement Learning for Neural Network Architecture Category : Theory & Reinforcement Learning Lei Lei Ruoxuan Xiong December 16, 2017 1 Introduction Deep Neural Network
Recurrent Neural Networks (RNN) and Long-Short-Term-Memory (LSTM) Yuan YAO HKUST
1 Recurrent Neural Networks (RNN) and Long-Short-Term-Memory (LSTM) Yuan YAO HKUST Summary We have shown: Now First order optimization methods: GD (BP), SGD, Nesterov, Adagrad, ADAM, RMSPROP, etc. Second
Recurrent Neural Networks. Jian Tang
Recurrent Neural Networks Jian Tang tangjianpku@gmail.com 1 RNN: Recurrent neural networks Neural networks for sequence modeling Summarize a sequence with fix-sized vector through recursively updating
Sequence Modeling with Neural Networks
Sequence Modeling with Neural Networks Harini Suresh y 0 y 1 y 2 s 0 s 1 s 2... x 0 x 1 x 2 hat is a sequence? This morning I took the dog for a walk. sentence medical signals speech waveform Successes
Neural Architectures for Image, Language, and Speech Processing
Neural Architectures for Image, Language, and Speech Processing Karl Stratos June 26, 2018 1 / 31 Overview Feedforward Networks Need for Specialized Architectures Convolutional Neural Networks (CNNs) Recurrent
Binary Convolutional Neural Network on RRAM
Binary Convolutional Neural Network on RRAM Tianqi Tang, Lixue Xia, Boxun Li, Yu Wang, Huazhong Yang Dept. of E.E, Tsinghua National Laboratory for Information Science and Technology (TNList) Tsinghua
Faster Machine Learning via Low-Precision Communication & Computation. Dan Alistarh (IST Austria & ETH Zurich), Hantian Zhang (ETH Zurich)
Faster Machine Learning via Low-Precision Communication & Computation Dan Alistarh (IST Austria & ETH Zurich), Hantian Zhang (ETH Zurich) 2 How many bits do you need to represent a single number in machine
Deep Learning Lecture 2
Fall 2016 Machine Learning CMPSCI 689 Deep Learning Lecture 2 Sridhar Mahadevan Autonomous Learning Lab UMass Amherst COLLEGE Outline of lecture New type of units Convolutional units, Rectified linear
arxiv: v1 [cs.ne] 20 Apr 2018
![arxiv: v1 [cs.ne] 20 Apr 2018](/thumbs/86/94149502.jpg "arxiv: v1 [cs.ne] 20 Apr 2018")
arxiv:1804.07802v1 [cs.ne] 20 Apr 2018 Value-aware Quantization for Training and Inference of Neural Networks Eunhyeok Park 1, Sungjoo Yoo 1, Peter Vajda 2 1 Department of Computer Science and Engineering
Lecture 5 Neural models for NLP
CS546: Machine Learning in NLP (Spring 2018) http://courses.engr.illinois.edu/cs546/ Lecture 5 Neural models for NLP Julia Hockenmaier juliahmr@illinois.edu 3324 Siebel Center Office hours: Tue/Thu 2pm-3pm
Conditional Language modeling with attention
Conditional Language modeling with attention 2017.08.25 Oxford Deep NLP 조수현 Review Conditional language model: assign probabilities to sequence of words given some conditioning context x What is the probability
CS260: Machine Learning Algorithms
CS260: Machine Learning Algorithms Lecture 4: Stochastic Gradient Descent Cho-Jui Hsieh UCLA Jan 16, 2019 Large-scale Problems Machine learning: usually minimizing the training loss min w { 1 N min w {
Memory-Augmented Attention Model for Scene Text Recognition
Memory-Augmented Attention Model for Scene Text Recognition Cong Wang 1,2, Fei Yin 1,2, Cheng-Lin Liu 1,2,3 1 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences
Outline. Overview CNTK introduction. Educational resources Conclusions. Symbolic loop Batch scheduling Data parallel training
Outline Overview CNTK introduction Symbolic loop Batch scheduling Data parallel training Educational resources Conclusions Outline Overview CNTK introduction Symbolic loop Batch scheduling Data parallel
Eve: A Gradient Based Optimization Method with Locally and Globally Adaptive Learning Rates
Eve: A Gradient Based Optimization Method with Locally and Globally Adaptive Learning Rates Hiroaki Hayashi 1,* Jayanth Koushik 1,* Graham Neubig 1 arxiv:1611.01505v3 [cs.lg] 11 Jun 2018 Abstract Adaptive
Intro to Neural Networks and Deep Learning
Intro to Neural Networks and Deep Learning Jack Lanchantin Dr. Yanjun Qi UVA CS 6316 1 Neurons 1-Layer Neural Network Multi-layer Neural Network Loss Functions Backpropagation Nonlinearity Functions NNs
Lecture 15: Exploding and Vanishing Gradients
Lecture 15: Exploding and Vanishing Gradients Roger Grosse 1 Introduction Last lecture, we introduced RNNs and saw how to derive the gradients using backprop through time. In principle, this lets us train
Pytorch Tutorial. Xiaoyong Yuan, Xiyao Ma 2018/01
(Li Lab) National Science Foundation Center for Big Learning (CBL) Department of Electrical and Computer Engineering (ECE) Department of Computer & Information Science & Engineering (CISE) Pytorch Tutorial
Asaf Bar Zvi Adi Hayat. Semantic Segmentation
Asaf Bar Zvi Adi Hayat Semantic Segmentation Today s Topics Fully Convolutional Networks (FCN) (CVPR 2015) Conditional Random Fields as Recurrent Neural Networks (ICCV 2015) Gaussian Conditional random
Supervised Learning in Neural Networks
The Norwegian University of Science and Technology (NTNU Trondheim, Norway keithd@idi.ntnu.no March 7, 2011 Supervised Learning Constant feedback from an instructor, indicating not only right/wrong, but
Value-aware Quantization for Training and Inference of Neural Networks
Value-aware Quantization for Training and Inference of Neural Networks Eunhyeok Park 1, Sungjoo Yoo 1, and Peter Vajda 2 1 Department of Computer Science and Engineering Seoul National University {eunhyeok.park,sungjoo.yoo}@gmail.com
Machine Learning for Large-Scale Data Analysis and Decision Making A. Neural Networks Week #6
Machine Learning for Large-Scale Data Analysis and Decision Making 80-629-17A Neural Networks Week #6 Today Neural Networks A. Modeling B. Fitting C. Deep neural networks Today s material is (adapted)
Bridging the Gap between Stochastic Gradient MCMC and Stochastic Optimization
Bridging the Gap between Stochastic Gradient MCMC and Stochastic Optimization Changyou Chen, David Carlson, Zhe Gan, Chunyuan Li, Lawrence Carin May 2, 2016 1 Changyou Chen Bridging the Gap between Stochastic
More Tips for Training Neural Network. Hung-yi Lee
More Tips for Training Neural Network Hung-yi ee Outline Activation Function Cost Function Data Preprocessing Training Generalization Review: Training Neural Network Neural network: f ; θ : input (vector)
Tasks ADAS. Self Driving. Non-machine Learning. Traditional MLP. Machine-Learning based method. Supervised CNN. Methods. Deep-Learning based
UNDERSTANDING CNN ADAS Tasks Self Driving Localizati on Perception Planning/ Control Driver state Vehicle Diagnosis Smart factory Methods Traditional Deep-Learning based Non-machine Learning Machine-Learning
Scalable Methods for 8-bit Training of Neural Networks
Scalable Methods for 8-bit Training of Neural Networks Ron Banner 1, Itay Hubara 2, Elad Hoffer 2, Daniel Soudry 2 {itayhubara, elad.hoffer, daniel.soudry}@gmail.com {ron.banner}@intel.com (1) Intel -
Serious limitations of (single-layer) perceptrons: Cannot learn non-linearly separable tasks. Cannot approximate (learn) non-linear functions
BACK-PROPAGATION NETWORKS Serious limitations of (single-layer) perceptrons: Cannot learn non-linearly separable tasks Cannot approximate (learn) non-linear functions Difficult (if not impossible) to design
Backpropagation: The Good, the Bad and the Ugly
Backpropagation: The Good, the Bad and the Ugly The Norwegian University of Science and Technology (NTNU Trondheim, Norway keithd@idi.ntnu.no October 3, 2017 Supervised Learning Constant feedback from
Lecture 35: Optimization and Neural Nets
Lecture 35: Optimization and Neural Nets CS 4670/5670 Sean Bell DeepDream [Google, Inceptionism: Going Deeper into Neural Networks, blog 2015] Aside: CNN vs ConvNet Note: There are many papers that use
CSC321 Lecture 15: Exploding and Vanishing Gradients
CSC321 Lecture 15: Exploding and Vanishing Gradients Roger Grosse Roger Grosse CSC321 Lecture 15: Exploding and Vanishing Gradients 1 / 23 Overview Yesterday, we saw how to compute the gradient descent
TTIC 31230, Fundamentals of Deep Learning David McAllester, April Vanishing and Exploding Gradients. ReLUs. Xavier Initialization
TTIC 31230, Fundamentals of Deep Learning David McAllester, April 2017 Vanishing and Exploding Gradients ReLUs Xavier Initialization Batch Normalization Highway Architectures: Resnets, LSTMs and GRUs Causes
Regularization and Optimization of Backpropagation
Regularization and Optimization of Backpropagation The Norwegian University of Science and Technology (NTNU) Trondheim, Norway keithd@idi.ntnu.no October 17, 2017 Regularization Definition of Regularization
Recurrent Neural Networks Deep Learning Lecture 5. Efstratios Gavves
Recurrent Neural Networks Deep Learning Lecture 5 Efstratios Gavves Sequential Data So far, all tasks assumed stationary data Neither all data, nor all tasks are stationary though Sequential Data: Text
Deep Neural Networks (3) Computational Graphs, Learning Algorithms, Initialisation
Deep Neural Networks (3) Computational Graphs, Learning Algorithms, Initialisation Steve Renals Machine Learning Practical MLP Lecture 5 16 October 2018 MLP Lecture 5 / 16 October 2018 Deep Neural Networks
TTIC 31230, Fundamentals of Deep Learning David McAllester, April Sequence to Sequence Models and Attention
TTIC 31230, Fundamentals of Deep Learning David McAllester, April 2017 Sequence to Sequence Models and Attention Encode-Decode Architectures for Machine Translation [Figure from Luong et al.] In Sutskever
Slide credit from Hung-Yi Lee & Richard Socher
Slide credit from Hung-Yi Lee & Richard Socher 1 Review Recurrent Neural Network 2 Recurrent Neural Network Idea: condition the neural network on all previous words and tie the weights at each time step
More on Neural Networks
More on Neural Networks Yujia Yan Fall 2018 Outline Linear Regression y = Wx + b (1) Linear Regression y = Wx + b (1) Polynomial Regression y = Wφ(x) + b (2) where φ(x) gives the polynomial basis, e.g.,
Lecture 7 Convolutional Neural Networks
Lecture 7 Convolutional Neural Networks CMSC 35246: Deep Learning Shubhendu Trivedi & Risi Kondor University of Chicago April 17, 2017 We saw before: ŷ x 1 x 2 x 3 x 4 A series of matrix multiplications:
Long-Short Term Memory and Other Gated RNNs
Long-Short Term Memory and Other Gated RNNs Sargur Srihari srihari@buffalo.edu This is part of lecture slides on Deep Learning: http://www.cedar.buffalo.edu/~srihari/cse676 1 Topics in Sequence Modeling
TTIC 31230, Fundamentals of Deep Learning, Winter David McAllester. The Fundamental Equations of Deep Learning
TTIC 31230, Fundamentals of Deep Learning, Winter 2019 David McAllester The Fundamental Equations of Deep Learning 1 Early History 1943: McCullock and Pitts introduced the linear threshold neuron. 1962:
Neural Networks 2. 2 Receptive fields and dealing with image inputs
CS 446 Machine Learning Fall 2016 Oct 04, 2016 Neural Networks 2 Professor: Dan Roth Scribe: C. Cheng, C. Cervantes Overview Convolutional Neural Networks Recurrent Neural Networks 1 Introduction There
CS 179: LECTURE 16 MODEL COMPLEXITY, REGULARIZATION, AND CONVOLUTIONAL NETS
CS 179: LECTURE 16 MODEL COMPLEXITY, REGULARIZATION, AND CONVOLUTIONAL NETS LAST TIME Intro to cudnn Deep neural nets using cublas and cudnn TODAY Building a better model for image classification Overfitting
Lecture 8: Introduction to Deep Learning: Part 2 (More on backpropagation, and ConvNets)
COS 402 Machine Learning and Artificial Intelligence Fall 2016 Lecture 8: Introduction to Deep Learning: Part 2 (More on backpropagation, and ConvNets) Sanjeev Arora Elad Hazan Recap: Structure of a deep
Deep Learning Recurrent Networks 2/28/2018
Deep Learning Recurrent Networks /8/8 Recap: Recurrent networks can be incredibly effective Story so far Y(t+) Stock vector X(t) X(t+) X(t+) X(t+) X(t+) X(t+5) X(t+) X(t+7) Iterated structures are good
Random Coattention Forest for Question Answering
Random Coattention Forest for Question Answering Jheng-Hao Chen Stanford University jhenghao@stanford.edu Ting-Po Lee Stanford University tingpo@stanford.edu Yi-Chun Chen Stanford University yichunc@stanford.edu
Tutorial on: Optimization I. (from a deep learning perspective) Jimmy Ba
Tutorial on: Optimization I (from a deep learning perspective) Jimmy Ba Outline Random search v.s. gradient descent Finding better search directions Design white-box optimization methods to improve computation
Deep Learning Year in Review 2016: Computer Vision Perspective
Deep Learning Year in Review 2016: Computer Vision Perspective Alex Kalinin, PhD Candidate Bioinformatics @ UMich alxndrkalinin@gmail.com @alxndrkalinin Architectures Summary of CNN architecture development
Advanced Training Techniques. Prajit Ramachandran
Advanced Training Techniques Prajit Ramachandran Outline Optimization Regularization Initialization Optimization Optimization Outline Gradient Descent Momentum RMSProp Adam Distributed SGD Gradient Noise
Identifying QCD transition using Deep Learning
Identifying QCD transition using Deep Learning Kai Zhou Long-Gang Pang, Nan Su, Hannah Peterson, Horst Stoecker, Xin-Nian Wang Collaborators: arxiv:1612.04262 Outline 2 What is deep learning? Artificial
The K-FAC method for neural network optimization
The K-FAC method for neural network optimization James Martens Thanks to my various collaborators on K-FAC research and engineering: Roger Grosse, Jimmy Ba, Vikram Tankasali, Matthew Johnson, Daniel Duckworth,
Speech and Language Processing
Speech and Language Processing Lecture 5 Neural network based acoustic and language models Information and Communications Engineering Course Takahiro Shinoaki 08//6 Lecture Plan (Shinoaki s part) I gives
Recurrent Autoregressive Networks for Online Multi-Object Tracking. Presented By: Ishan Gupta
Recurrent Autoregressive Networks for Online Multi-Object Tracking Presented By: Ishan Gupta Outline Multi Object Tracking Recurrent Autoregressive Networks (RANs) RANs for Online Tracking Other State
Hardware Acceleration of DNNs
Lecture 12: Hardware Acceleration of DNNs Visual omputing Systems Stanford S348V, Winter 2018 Hardware acceleration for DNNs Huawei Kirin NPU Google TPU: Apple Neural Engine Intel Lake rest Deep Learning
Based on the original slides of Hung-yi Lee
Based on the original slides of Hung-yi Lee New Activation Function Rectified Linear Unit (ReLU) σ z a a = z Reason: 1. Fast to compute 2. Biological reason a = 0 [Xavier Glorot, AISTATS 11] [Andrew L.
Natural Language Processing
Natural Language Processing Pushpak Bhattacharyya CSE Dept, IIT Patna and Bombay LSTM 15 jun, 2017 lgsoft:nlp:lstm:pushpak 1 Recap 15 jun, 2017 lgsoft:nlp:lstm:pushpak 2 Feedforward Network and Backpropagation
Machine Learning for Signal Processing Neural Networks Continue. Instructor: Bhiksha Raj Slides by Najim Dehak 1 Dec 2016
Machine Learning for Signal Processing Neural Networks Continue Instructor: Bhiksha Raj Slides by Najim Dehak 1 Dec 2016 1 So what are neural networks?? Voice signal N.Net Transcription Image N.Net Text
Lecture 6 Optimization for Deep Neural Networks
Lecture 6 Optimization for Deep Neural Networks CMSC 35246: Deep Learning Shubhendu Trivedi & Risi Kondor University of Chicago April 12, 2017 Things we will look at today Stochastic Gradient Descent Things
Understanding How ConvNets See
Understanding How ConvNets See Slides from Andrej Karpathy Springerberg et al, Striving for Simplicity: The All Convolutional Net (ICLR 2015 workshops) CSC321: Intro to Machine Learning and Neural Networks,
arxiv: v1 [cs.cv] 11 May 2015 Abstract
![arxiv: v1 [cs.cv] 11 May 2015 Abstract](/thumbs/80/80674557.jpg "arxiv: v1 [cs.cv] 11 May 2015 Abstract")
Training Deeper Convolutional Networks with Deep Supervision Liwei Wang Computer Science Dept UIUC lwang97@illinois.edu Chen-Yu Lee ECE Dept UCSD chl260@ucsd.edu Zhuowen Tu CogSci Dept UCSD ztu0@ucsd.edu
DEEP LEARNING AND NEURAL NETWORKS: BACKGROUND AND HISTORY
DEEP LEARNING AND NEURAL NETWORKS: BACKGROUND AND HISTORY 1 On-line Resources http://neuralnetworksanddeeplearning.com/index.html Online book by Michael Nielsen http://matlabtricks.com/post-5/3x3-convolution-kernelswith-online-demo
HIGH PERFORMANCE CTC TRAINING FOR END-TO-END SPEECH RECOGNITION ON GPU
April 4-7, 2016 Silicon Valley HIGH PERFORMANCE CTC TRAINING FOR END-TO-END SPEECH RECOGNITION ON GPU Minmin Sun, NVIDIA minmins@nvidia.com April 5th Brief Introduction of CTC AGENDA Alpha/Beta Matrix
TYPES OF MODEL COMPRESSION. Soham Saha, MS by Research in CSE, CVIT, IIIT Hyderabad
TYPES OF MODEL COMPRESSION Soham Saha, MS by Research in CSE, CVIT, IIIT Hyderabad 1. Pruning 2. Quantization 3. Architectural Modifications PRUNING WHY PRUNING? Deep Neural Networks have redundant parameters.
Index. Santanu Pattanayak 2017 S. Pattanayak, Pro Deep Learning with TensorFlow,
Index A Activation functions, neuron/perceptron binary threshold activation function, 102 103 linear activation function, 102 rectified linear unit, 106 sigmoid activation function, 103 104 SoftMax activation
High Order LSTM/GRU. Wenjie Luo. January 19, 2016
High Order LSTM/GRU Wenjie Luo January 19, 2016 1 Introduction RNN is a powerful model for sequence data but suffers from gradient vanishing and explosion, thus difficult to be trained to capture long
A Practitioner s Guide to MXNet
1/34 A Practitioner s Guide to MXNet Xingjian Shi Hong Kong University of Science and Technology (HKUST) HKUST CSE Seminar, March 31st, 2017 2/34 Outline 1 Introduction Deep Learning Basics MXNet Highlights
Agenda. Digit Classification using CNN Digit Classification using SAE Visualization: Class models, filters and saliency 2 DCT
versus 1 Agenda Deep Learning: Motivation Learning: Backpropagation Deep architectures I: Convolutional Neural Networks (CNN) Deep architectures II: Stacked Auto Encoders (SAE) Caffe Deep Learning Toolbox:
Deep Residual. Variations
Deep Residual Network and Its Variations Diyu Yang (Originally prepared by Kaiming He from Microsoft Research) Advantages of Depth Degradation Problem Possible Causes? Vanishing/Exploding Gradients. Overfitting
Deep learning attracts lots of attention.
Deep Learning Deep learning attracts lots of attention. I believe you have seen lots of exciting results before. Deep learning trends at Google. Source: SIGMOD/Jeff Dean Ups and downs of Deep Learning
CNTK Microsoft s Open Source Deep Learning Toolkit. Taifeng Wang Lead Researcher, Microsoft Research Asia 2016 GTC China
CNTK Microsoft s Open Source Deep Learning Toolkit Taifeng Wang Lead Researcher, Microsoft Research Asia 2016 GTC China Deep learning in Microsoft Cognitive Services https://how-old.net http://www.captionbot.ai
CSC321 Lecture 20: Reversible and Autoregressive Models
CSC321 Lecture 20: Reversible and Autoregressive Models Roger Grosse Roger Grosse CSC321 Lecture 20: Reversible and Autoregressive Models 1 / 23 Overview Four modern approaches to generative modeling:
RECURRENT NETWORKS I. Philipp Krähenbühl
RECURRENT NETWORKS I Philipp Krähenbühl RECAP: CLASSIFICATION conv 1 conv 2 conv 3 conv 4 1 2 tu RECAP: SEGMENTATION conv 1 conv 2 conv 3 conv 4 RECAP: DETECTION conv 1 conv 2 conv 3 conv 4 RECAP: GENERATION
Classification goals: Make 1 guess about the label (Top-1 error) Make 5 guesses about the label (Top-5 error) No Bounding Box
ImageNet Classification with Deep Convolutional Neural Networks Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton Motivation Classification goals: Make 1 guess about the label (Top-1 error) Make 5 guesses
Improved Learning through Augmenting the Loss
Improved Learning through Augmenting the Loss Hakan Inan inanh@stanford.edu Khashayar Khosravi khosravi@stanford.edu Abstract We present two improvements to the well-known Recurrent Neural Network Language
Recurrent Neural Network
Recurrent Neural Network Xiaogang Wang xgwang@ee..edu.hk March 2, 2017 Xiaogang Wang (linux) Recurrent Neural Network March 2, 2017 1 / 48 Outline 1 Recurrent neural networks Recurrent neural networks
Stephen Scott.
1 / 35 (Adapted from Vinod Variyam and Ian Goodfellow) sscott@cse.unl.edu 2 / 35 All our architectures so far work on fixed-sized inputs neural networks work on sequences of inputs E.g., text, biological
CSC321 Lecture 9: Generalization
CSC321 Lecture 9: Generalization Roger Grosse Roger Grosse CSC321 Lecture 9: Generalization 1 / 26 Overview We ve focused so far on how to optimize neural nets how to get them to make good predictions
CSCI 1951-G Optimization Methods in Finance Part 12: Variants of Gradient Descent
CSCI 1951-G Optimization Methods in Finance Part 12: Variants of Gradient Descent April 27, 2018 1 / 32 Outline 1) Moment and Nesterov s accelerated gradient descent 2) AdaGrad and RMSProp 4) Adam 5) Stochastic
What Do Neural Networks Do? MLP Lecture 3 Multi-layer networks 1
What Do Neural Networks Do? MLP Lecture 3 Multi-layer networks 1 Multi-layer networks Steve Renals Machine Learning Practical MLP Lecture 3 7 October 2015 MLP Lecture 3 Multi-layer networks 2 What Do Single
Distributed Machine Learning: A Brief Overview. Dan Alistarh IST Austria
Distributed Machine Learning: A Brief Overview Dan Alistarh IST Austria Background The Machine Learning Cambrian Explosion Key Factors: 1. Large s: Millions of labelled images, thousands of hours of speech
Neural Networks. David Rosenberg. July 26, New York University. David Rosenberg (New York University) DS-GA 1003 July 26, / 35
Neural Networks David Rosenberg New York University July 26, 2017 David Rosenberg (New York University) DS-GA 1003 July 26, 2017 1 / 35 Neural Networks Overview Objectives What are neural networks? How
Faster Training of Very Deep Networks Via p-norm Gates
Faster Training of Very Deep Networks Via p-norm Gates Trang Pham, Truyen Tran, Dinh Phung, Svetha Venkatesh Center for Pattern Recognition and Data Analytics Deakin University, Geelong Australia Email:
Deep Learning. Hung-yi Lee 李宏毅
Deep Learning Hung-yi Lee 李宏毅 Deep learning attracts lots of attention. I believe you have seen lots of exciting results before. Deep learning trends at Google. Source: SIGMOD 206/Jeff Dean 958: Perceptron
Supervised Deep Learning
Supervised Deep Learning Joana Frontera-Pons Grigorios Tsagkatakis Dictionary Learning on Manifolds workshop Nice, September 2017 Supervised Learning Data Labels Model Prediction Spiral Exploiting prior
PRUNING CONVOLUTIONAL NEURAL NETWORKS. Pavlo Molchanov Stephen Tyree Tero Karras Timo Aila Jan Kautz
PRUNING CONVOLUTIONAL NEURAL NETWORKS Pavlo Molchanov Stephen Tyree Tero Karras Timo Aila Jan Kautz 2017 WHY WE CAN PRUNE CNNS? 2 WHY WE CAN PRUNE CNNS? Optimization failures : Some neurons are "dead":