Convolutional Neural Networks
|
|
|
- Jessie Sutton
- 5 years ago
- Views:
Transcription
1 Convolutional Neural Networks
2 Books»
3 Books
4 reviews»
5
6 Goal: Given an image, we want to identify what class that image belongs to. Input: Output:
7 Pipeline: Input Convolutional Neural Network (CNN) Output A Monitor
8 Convolutional Neural Nets (CNNs) in a nutshell: A typical CNN takes a raw RGB image as an input. It then applies a series of non-linear operations on top of each other. These include convolution, sigmoid, matrix multiplication, and pooling (subsampling) operations. The output of a CNN is a highly non-linear function of the raw RGB image pixels.
9 How the key operations are encoded in standard CNNs: Convolutional Layers: 2D Convolution Fully Connected Layers: Matrix Multiplication Sigmoid Layers: Sigmoid function Pooling Layers: Subsampling
10 2D convolution: h f g - the values in a 2D grid that we want to convolve - convolutional weights of size MxN M N hij f ( i m, j n) g( m, n) m 0 n 0 A sliding window operation across the entire grid.
11 f g2 3 f g 1 f g Unchanged Image Blurred Image Vertical Edges
12 CNNs aim to learn convolutional weights directly from the data
13 Input: Convolutional Neural Network (CNN) Early layers learn to detect low level structures such as oriented edges, colors and corners
14 Input: Convolutional Neural Network (CNN) Deep layers learn to detect high-level object structures and their parts.
15 A Closer Look inside the Convolutional Layer A Chair Filter Input Image A Person Filter A Table Filter A Cupboard Filter
16 Fully Connected Layers:
17 Fully Connected Layers: matrix multiplication
18 Max Pooling Layer: Sliding window is applied on a grid of values. The maximum is computed using the values in the current window
19 Max Pooling Layer: Sliding window is applied on a grid of values. The maximum is computed using the values in the current window
20 Max Pooling Layer: Sliding window is applied on a grid of values. The maximum is computed using the values in the current window
21 Max Pooling Layer: Sliding window is applied on a grid of values. The maximum is computed using the values in the current window
22 Sigmoid Layer: Applies a sigmoid function on an input
23 Convolutional Networks Let us now consider a CNN with a specific architecture: 2 convolutional layers. 2 pooling layers. 2 fully connected layers. 3 sigmoid layers.
24 Notation: - convolutional layer output - fully connected layer output - pooling layer - sigmoid function - softmax function Forward Pass:
25 Notation: - convolutional layer output - fully connected layer output - pooling layer - sigmoid function - softmax function Forward Pass:
26 Notation: - convolutional layer output - fully connected layer output - pooling layer - sigmoid function - softmax function Forward Pass:
27 Notation: - convolutional layer output - fully connected layer output - pooling layer - sigmoid function - softmax function Forward Pass:
28 Notation: - convolutional layer output - fully connected layer output - pooling layer - sigmoid function - softmax function Forward Pass:
29 Notation: - convolutional layer output - fully connected layer output - pooling layer - sigmoid function - softmax function Forward Pass:
30 Notation: - convolutional layer output - fully connected layer output - pooling layer - sigmoid function - softmax function Forward Pass:
31 Notation: - convolutional layer output - fully connected layer output - pooling layer - sigmoid function - softmax function Forward Pass:
32 Notation: - convolutional layer output - fully connected layer output - pooling layer - sigmoid function - softmax function Forward Pass:
33 Convolutional Networks Notation: - convolutional layer output - fully connected layer output - pooling layer - sigmoid function - softmax function Forward Pass:
34 Notation: - convolutional layer output - fully connected layer output - pooling layer - sigmoid function - softmax function Forward Pass:
35 Notation: - convolutional layer output - fully connected layer output - pooling layer - sigmoid function - softmax function Forward Pass: Final Predictions
36 Notation: - convolutional layer output - fully connected layer output - pooling layer - sigmoid function - softmax function Forward Pass: Convolutional layer parameters in layers 1 and 2
37 Notation: - convolutional layer output - fully connected layer output - pooling layer - sigmoid function - softmax function Forward Pass: Fully connected layer parameters in the fully connected layers 1 and 2
38 Notation: - convolutional layer output - fully connected layer output - pooling layer - sigmoid function - softmax function Forward Pass:
39 Notation: - convolutional layer output - fully connected layer output - pooling layer - sigmoid function - softmax function Forward Pass: 1.
40 Notation: - convolutional layer output - fully connected layer output - pooling layer - sigmoid function - softmax function Forward Pass: 1. 2.
41 Notation: - convolutional layer output - fully connected layer output - pooling layer - sigmoid function - softmax function Forward Pass:
42 Notation: - convolutional layer output - fully connected layer output - pooling layer - sigmoid function - softmax function Forward Pass:
43 Notation: - convolutional layer output - fully connected layer output - pooling layer - sigmoid function - softmax function Forward Pass: Key Question: How to learn the parameters from the data?
44 Backpropagation for Convolutional Neural Networks
45 Prediction True Label How to learn the parameters of a CNN? Assume that we are a given a labeled training dataset We want to adjust the parameters of a CNN such that CNN s predictions would be as close to true labels as possible. This is difficult to do because the learning objective is highly non-linear.
46 Gradient descent: Iteratively minimizes the objective function. The function needs to be differentiable.
47 Gradient descent: Iteratively minimizes the objective function. The function needs to be differentiable.
48 Gradient descent: Iteratively minimizes the objective function. The function needs to be differentiable.
49 Gradient descent: Iteratively minimizes the objective function. The function needs to be differentiable.
50 1. Compute the gradients of the overall loss w.r.t. to our predictions and propagate it back: True Label
51 2. Compute the gradients of the overall loss and propagate it back: True Label
52 True Label 3. Compute the gradients to adjust the weights:
53 4. Backpropagate the gradients to previous layers: True Label
54 True Label 5. Compute the gradients to adjust the weights:
55 6. Backpropagate the gradients to previous layers: True Label
56 True Label 7. Compute the gradients to adjust the weights:
57 8. Backpropagate the gradients to previous layers: True Label
58 True Label 9. Compute the gradients to adjust the weights:
59 An Example of Backpropagation Convolutional Neural Networks
60 True Label Assume that we have K=5 object classes: Class 1: Penguin Class 2: Building Class 3: Chair Class 4: Person Class 5: Bird
61 True Label
62 where True Label
63 where True Label
64 where True Label
65 where True Label
66 True Label
67 True Label
68 True Label
69 True Label Assume that we have K=5 object classes: Class 1: Penguin Class 2: Building Class 3: Chair Class 4: Person Class 5: Bird
70 True Label Assume that we have K=5 object classes: Class 1: Penguin Class 2: Building Class 3: Chair Class 4: Person Class 5: Bird
71 True Label Assume that we have K=5 object classes: Class 1: Penguin Class 2: Building Class 3: Chair Class 4: Person Class 5: Bird Increasing the score corresponding to the true class decreases the loss.
72 True Label Assume that we have K=5 object classes: Class 1: Penguin Class 2: Building Class 3: Chair Class 4: Person Class 5: Bird Decreasing the score of other classes also decreases the loss.
73 Adjusting the weights: True Label
74 True Label Adjusting the weights: Need to compute the following gradient
75 True Label Adjusting the weights: Need to compute the following gradient
76 True Label Adjusting the weights: Need to compute the following gradient was already computed in the previous step
77 True Label Adjusting the weights: Need to compute the following gradient
78 True Label Adjusting the weights: Need to compute the following gradient where
79 True Label Adjusting the weights: Need to compute the following gradient where
80 True Label Adjusting the weights: Need to compute the following gradient Update rule:
81 Backpropagating the gradients: True Label
82 True Label Backpropagating the gradients: Need to compute the following gradient:
83 True Label Backpropagating the gradients: Need to compute the following gradient:
84 True Label Backpropagating the gradients: Need to compute the following gradient: was already computed in the previous step
85 True Label Backpropagating the gradients: Need to compute the following gradient:
86 True Label Backpropagating the gradients: Need to compute the following gradient: where
87 True Label Backpropagating the gradients: Need to compute the following gradient: where
88 True Label Backpropagating the gradients: Need to compute the following gradient:
89 True Label Backpropagating the gradients: Need to compute the following gradient:
90 True Label Backpropagating the gradients: Need to compute the following gradient:
91 Adjusting the weights: True Label
92 True Label Adjusting the weights: Need to compute the following gradient Update rule:
93 Backpropagating the gradients: True Label
94 True Label Backpropagating the gradients: Need to compute the following gradient:
95 Adjusting the weights: True Label
96 True Label Adjusting the weights: Need to compute the following gradient
97 True Label Adjusting the weights: Need to compute the following gradient
98 True Label Adjusting the weights: Need to compute the following gradient was already computed in the previous step
99 True Label Adjusting the weights: Need to compute the following gradient
100 True Label Adjusting the weights: Need to compute the following gradient where
101 True Label Adjusting the weights: Need to compute the following gradient where
102 True Label Adjusting the weights: Need to compute the following gradient Update rule:
103 Backpropagating the gradients: True Label
104 True Label Backpropagating the gradients: Need to compute the following gradient:
105 True Label Backpropagating the gradients: Need to compute the following gradient:
106 True Label Backpropagating the gradients: Need to compute the following gradient: was already computed in the previous step
107 True Label Backpropagating the gradients: Need to compute the following gradient:
108 True Label Backpropagating the gradients: Need to compute the following gradient: where
109 True Label Backpropagating the gradients: Need to compute the following gradient: where
110 True Label Backpropagating the gradients: Need to compute the following gradient:
111 True Label Backpropagating the gradients: Need to compute the following gradient:
112 True Label Backpropagating the gradients: Need to compute the following gradient:
113 Adjusting the weights: True Label
114 True Label Adjusting the weights: Need to compute the following gradient Update rule:
115 Visual illustration Backpropagation Convolutional Neural Networks
116 Fully Connected Layers: Forward:
117 Fully Connected Layers: Forward: Activation unit of interest
118 Fully Connected Layers: Forward: Activation unit of interest The output The weight that is used in conjunction with the activation unit of interest
119 Fully Connected Layers: Forward:
120 Fully Connected Layers: Forward:
121 Fully Connected Layers: Forward:
122 Fully Connected Layers: Forward: Backpropagation
123 Fully Connected Layers: Forward:
124 Fully Connected Layers: Forward:
125 Fully Connected Layers: Forward:
126 Fully Connected Layers: Forward: Backward:
127 Fully Connected Layers: Forward: A measure how much an activation unit contributed to the loss Backward:
128 Fully Connected Layers: Forward: A measure how much an activation unit contributed to the loss Backward:
129 Fully Connected Layers: Forward: Backward:
130 Fully Connected Layers: Forward: Backward:
131 Fully Connected Layers: Forward: Backward:
132 Fully Connected Layers: Forward: Backward:
133 Fully Connected Layers: Forward: Backward:
134 Fully Connected Layers: Forward: Backward:
135 Summary for fully connected layers Backpropagation Convolutional Neural Networks
136 Summary: 1. Let, where n denotes the number of layers in the network.
137 Summary: 1. Let, where n denotes the number of layers in the network. 2. For each fully connected layer : For each node in layer set:
138 Summary: 1. Let, where n denotes the number of layers in the network. 2. For each fully connected layer : For each node in layer set: Compute partial derivatives:
139 Summary: 1. Let, where n denotes the number of layers in the network. 2. For each fully connected layer : For each node in layer set: Compute partial derivatives: Update the parameters:
140 Visual illustration Backpropagation Convolutional Neural Networks
141 Convolutional Layers: Forward: a g z ( l ) ( l ) ( l 1 )
142 Convolutional Layers: Forward: a g z ( l ) ( l ) ( l 1 )
143 Convolutional Layers: Forward: a g z ( l ) ( l ) ( l 1 )
144 Summary: 1. Let, where c denotes the index of a first fully connected layer. 2. For each convolutional layer : For each node in layer set
145 Summary: 1. Let, where c denotes the index of a first fully connected layer. 2. For each convolutional layer : For each node in layer set Compute partial derivatives:
146 Summary: 1. Let, where c denotes the index of a first fully connected layer. 2. For each convolutional layer : For each node in layer set Compute partial derivatives: Update the parameters:
147 True Label Gradient in pooling layers: There is no learning done in the pooling layers The error that is backpropagated to the pooling layer, is sent back from to the node where it came from.
148 True Label Gradient in pooling layers: There is no learning done in the pooling layers The error that is backpropagated to the pooling layer, is sent back from to the node where it came from. Forward Pass Layer Layer
149 True Label Gradient in pooling layers: There is no learning done in the pooling layers The error that is backpropagated to the pooling layer, is sent back from to the node where it came from. Backward Pass Layer Layer 0 0 0
150 Recognition as a big table lookup Image bits Recognition Label n Image = Y N m N Y Y N Y write down all images in a table and record the object label Y N Y n*m N
151 How many images are there? An tiny image example (can you see what it is?) 3 18 Each pixel has 2 = 256 values 8 3x18 image above has 54 pixels Total possible 54 pixel images = = 1.1 x Prof. Kanade s Theorem: we have not seen anything yet!
152 3 Total possible 54 pixel images =1.1 x Compared number of images seen by all humans ever: 10 billion x 1000 x 100 x 356 x 24 x 60 x 60 x 30 = Total population years hours frame rate generations days min/sec We have to be clever in writing down this table!
153 A Closer Look inside the Convolutional Layer A Chair Filter Input Image A Person Filter A Table Filter A Cupboard Filter
154 A Closer Look inside the Convolutional Layer :
155 A Closer Look inside the Back Propagation Convolutional Layer :
156 Adjusting the weights: True Label
157 Training Batch ( ) ( ) ( ) ( ) Average the Gradient Across the Batch Parameter Update ( ) (performed in a sliding window fashion) - elementwise multiplication new old
158 Training Batch ( ) ( ) ( ) ( ) Average the Gradient Across the Batch Parameter Update ( ) (performed in a sliding window fashion) - elementwise multiplication new old
159 Training Batch ( ) ( ) ( ) ( ) Average the Gradient Across the Batch Parameter Update ( ) (performed in a sliding window fashion) - elementwise multiplication new old
160 Adjusting the weights: True Label
161 Training Batch ( ) ( ) ( ) ( ) Average the Gradient Across the Batch Parameter Update ( ) (performed in a sliding window fashion) - elementwise multiplication new old
162 Training Batch ( ) ( ) ( ) ( ) Average the Gradient Across the Batch Parameter Update ( ) (performed in a sliding window fashion) - elementwise multiplication new old
163 Training Batch ( ) ( ) ( ) ( ) Average the Gradient Across the Batch Parameter Update ( ) (performed in a sliding window fashion) - elementwise multiplication new old
Artificial Neural Networks D B M G. Data Base and Data Mining Group of Politecnico di Torino. Elena Baralis. Politecnico di Torino
Artificial Neural Networks Data Base and Data Mining Group of Politecnico di Torino Elena Baralis Politecnico di Torino Artificial Neural Networks Inspired to the structure of the human brain Neurons as
Classification goals: Make 1 guess about the label (Top-1 error) Make 5 guesses about the label (Top-5 error) No Bounding Box
ImageNet Classification with Deep Convolutional Neural Networks Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton Motivation Classification goals: Make 1 guess about the label (Top-1 error) Make 5 guesses
Lecture 17: Neural Networks and Deep Learning
UVA CS 6316 / CS 4501-004 Machine Learning Fall 2016 Lecture 17: Neural Networks and Deep Learning Jack Lanchantin Dr. Yanjun Qi 1 Neurons 1-Layer Neural Network Multi-layer Neural Network Loss Functions
Deep Learning (CNNs)
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Deep Learning (CNNs) Deep Learning Readings: Murphy 28 Bishop - - HTF - - Mitchell
ECE521 Lectures 9 Fully Connected Neural Networks
ECE521 Lectures 9 Fully Connected Neural Networks Outline Multi-class classification Learning multi-layer neural networks 2 Measuring distance in probability space We learnt that the squared L2 distance
Understanding How ConvNets See
Understanding How ConvNets See Slides from Andrej Karpathy Springerberg et al, Striving for Simplicity: The All Convolutional Net (ICLR 2015 workshops) CSC321: Intro to Machine Learning and Neural Networks,
Νεςπο-Ασαυήρ Υπολογιστική Neuro-Fuzzy Computing
Νεςπο-Ασαυήρ Υπολογιστική Neuro-Fuzzy Computing ΗΥ418 Διδάσκων Δημήτριος Κατσαρός @ Τμ. ΗΜΜΥ Πανεπιστήμιο Θεσσαλίαρ Διάλεξη 21η BackProp for CNNs: Do I need to understand it? Why do we have to write the
CS 179: LECTURE 16 MODEL COMPLEXITY, REGULARIZATION, AND CONVOLUTIONAL NETS
CS 179: LECTURE 16 MODEL COMPLEXITY, REGULARIZATION, AND CONVOLUTIONAL NETS LAST TIME Intro to cudnn Deep neural nets using cublas and cudnn TODAY Building a better model for image classification Overfitting
<Special Topics in VLSI> Learning for Deep Neural Networks (Back-propagation)
Learning for Deep Neural Networks (Back-propagation) Outline Summary of Previous Standford Lecture Universal Approximation Theorem Inference vs Training Gradient Descent Back-Propagation
(Feed-Forward) Neural Networks Dr. Hajira Jabeen, Prof. Jens Lehmann
(Feed-Forward) Neural Networks 2016-12-06 Dr. Hajira Jabeen, Prof. Jens Lehmann Outline In the previous lectures we have learned about tensors and factorization methods. RESCAL is a bilinear model for
Lecture 8: Introduction to Deep Learning: Part 2 (More on backpropagation, and ConvNets)
COS 402 Machine Learning and Artificial Intelligence Fall 2016 Lecture 8: Introduction to Deep Learning: Part 2 (More on backpropagation, and ConvNets) Sanjeev Arora Elad Hazan Recap: Structure of a deep
Neural Networks 2. 2 Receptive fields and dealing with image inputs
CS 446 Machine Learning Fall 2016 Oct 04, 2016 Neural Networks 2 Professor: Dan Roth Scribe: C. Cheng, C. Cervantes Overview Convolutional Neural Networks Recurrent Neural Networks 1 Introduction There
Neural Networks. Intro to AI Bert Huang Virginia Tech
Neural Networks Intro to AI Bert Huang Virginia Tech Outline Biological inspiration for artificial neural networks Linear vs. nonlinear functions Learning with neural networks: back propagation https://en.wikipedia.org/wiki/neuron#/media/file:chemical_synapse_schema_cropped.jpg
Neural Networks, Computation Graphs. CMSC 470 Marine Carpuat
Neural Networks, Computation Graphs CMSC 470 Marine Carpuat Binary Classification with a Multi-layer Perceptron φ A = 1 φ site = 1 φ located = 1 φ Maizuru = 1 φ, = 2 φ in = 1 φ Kyoto = 1 φ priest = 0 φ
Deep Learning. Convolutional Neural Network (CNNs) Ali Ghodsi. October 30, Slides are partially based on Book in preparation, Deep Learning
Convolutional Neural Network (CNNs) University of Waterloo October 30, 2015 Slides are partially based on Book in preparation, by Bengio, Goodfellow, and Aaron Courville, 2015 Convolutional Networks Convolutional
An overview of deep learning methods for genomics
An overview of deep learning methods for genomics Matthew Ploenzke STAT115/215/BIO/BIST282 Harvard University April 19, 218 1 Snapshot 1. Brief introduction to convolutional neural networks What is deep
Machine Learning for Large-Scale Data Analysis and Decision Making A. Neural Networks Week #6
Machine Learning for Large-Scale Data Analysis and Decision Making 80-629-17A Neural Networks Week #6 Today Neural Networks A. Modeling B. Fitting C. Deep neural networks Today s material is (adapted)
Neural Networks Learning the network: Backprop , Fall 2018 Lecture 4
Neural Networks Learning the network: Backprop 11-785, Fall 2018 Lecture 4 1 Recap: The MLP can represent any function The MLP can be constructed to represent anything But how do we construct it? 2 Recap:
Lecture 35: Optimization and Neural Nets
Lecture 35: Optimization and Neural Nets CS 4670/5670 Sean Bell DeepDream [Google, Inceptionism: Going Deeper into Neural Networks, blog 2015] Aside: CNN vs ConvNet Note: There are many papers that use
Spatial Transformer Networks
BIL722 - Deep Learning for Computer Vision Spatial Transformer Networks Max Jaderberg Andrew Zisserman Karen Simonyan Koray Kavukcuoglu Contents Introduction to Spatial Transformers Related Works Spatial
Machine Learning for Computer Vision 8. Neural Networks and Deep Learning. Vladimir Golkov Technical University of Munich Computer Vision Group
Machine Learning for Computer Vision 8. Neural Networks and Deep Learning Vladimir Golkov Technical University of Munich Computer Vision Group INTRODUCTION Nonlinear Coordinate Transformation http://cs.stanford.edu/people/karpathy/convnetjs/
RegML 2018 Class 8 Deep learning
RegML 2018 Class 8 Deep learning Lorenzo Rosasco UNIGE-MIT-IIT June 18, 2018 Supervised vs unsupervised learning? So far we have been thinking of learning schemes made in two steps f(x) = w, Φ(x) F, x
Deep Learning Recurrent Networks 2/28/2018
Deep Learning Recurrent Networks /8/8 Recap: Recurrent networks can be incredibly effective Story so far Y(t+) Stock vector X(t) X(t+) X(t+) X(t+) X(t+) X(t+5) X(t+) X(t+7) Iterated structures are good
Multilayer Neural Networks. (sometimes called Multilayer Perceptrons or MLPs)
Multilayer Neural Networks (sometimes called Multilayer Perceptrons or MLPs) Linear separability Hyperplane In 2D: w x + w 2 x 2 + w 0 = 0 Feature x 2 = w w 2 x w 0 w 2 Feature 2 A perceptron can separate
BACKPROPAGATION. Neural network training optimization problem. Deriving backpropagation
BACKPROPAGATION Neural network training optimization problem min J(w) w The application of gradient descent to this problem is called backpropagation. Backpropagation is gradient descent applied to J(w)
Deep Learning Lab Course 2017 (Deep Learning Practical)
Deep Learning Lab Course 207 (Deep Learning Practical) Labs: (Computer Vision) Thomas Brox, (Robotics) Wolfram Burgard, (Machine Learning) Frank Hutter, (Neurorobotics) Joschka Boedecker University of
Neural networks and optimization
Neural networks and optimization Nicolas Le Roux INRIA 8 Nov 2011 Nicolas Le Roux (INRIA) Neural networks and optimization 8 Nov 2011 1 / 80 1 Introduction 2 Linear classifier 3 Convolutional neural networks
Classification of Hand-Written Digits Using Scattering Convolutional Network
Mid-year Progress Report Classification of Hand-Written Digits Using Scattering Convolutional Network Dongmian Zou Advisor: Professor Radu Balan Co-Advisor: Dr. Maneesh Singh (SRI) Background Overview
Convolutional Neural Networks. Srikumar Ramalingam
Convolutional Neural Networks Srikumar Ramalingam Reference Many of the slides are prepared using the following resources: neuralnetworksanddeeplearning.com (mainly Chapter 6) http://cs231n.github.io/convolutional-networks/
Need for Deep Networks Perceptron. Can only model linear functions. Kernel Machines. Non-linearity provided by kernels
Need for Deep Networks Perceptron Can only model linear functions Kernel Machines Non-linearity provided by kernels Need to design appropriate kernels (possibly selecting from a set, i.e. kernel learning)
Agenda. Digit Classification using CNN Digit Classification using SAE Visualization: Class models, filters and saliency 2 DCT
versus 1 Agenda Deep Learning: Motivation Learning: Backpropagation Deep architectures I: Convolutional Neural Networks (CNN) Deep architectures II: Stacked Auto Encoders (SAE) Caffe Deep Learning Toolbox:
Logistic Regression & Neural Networks
Logistic Regression & Neural Networks CMSC 723 / LING 723 / INST 725 Marine Carpuat Slides credit: Graham Neubig, Jacob Eisenstein Logistic Regression Perceptron & Probabilities What if we want a probability
Convolution arithmetic
Institut des algorithmes d apprentissage de Montréal Convolution arithmetic Vincent Dumoulin Francesco Visin Introduction Why should I care? Convolutional neural networks are not a new idea (LeCun et al.
Neural Networks. Bishop PRML Ch. 5. Alireza Ghane. Feed-forward Networks Network Training Error Backpropagation Applications
Neural Networks Bishop PRML Ch. 5 Alireza Ghane Neural Networks Alireza Ghane / Greg Mori 1 Neural Networks Neural networks arise from attempts to model human/animal brains Many models, many claims of
Need for Deep Networks Perceptron. Can only model linear functions. Kernel Machines. Non-linearity provided by kernels
Need for Deep Networks Perceptron Can only model linear functions Kernel Machines Non-linearity provided by kernels Need to design appropriate kernels (possibly selecting from a set, i.e. kernel learning)
CS 1674: Intro to Computer Vision. Final Review. Prof. Adriana Kovashka University of Pittsburgh December 7, 2016
CS 1674: Intro to Computer Vision Final Review Prof. Adriana Kovashka University of Pittsburgh December 7, 2016 Final info Format: multiple-choice, true/false, fill in the blank, short answers, apply an
6.036: Midterm, Spring Solutions
6.036: Midterm, Spring 2018 Solutions This is a closed book exam. Calculators not permitted. The problems are not necessarily in any order of difficulty. Record all your answers in the places provided.
Bayesian Networks (Part I)
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Bayesian Networks (Part I) Graphical Model Readings: Murphy 10 10.2.1 Bishop 8.1,
Feed-forward Networks Network Training Error Backpropagation Applications. Neural Networks. Oliver Schulte - CMPT 726. Bishop PRML Ch.
Neural Networks Oliver Schulte - CMPT 726 Bishop PRML Ch. 5 Neural Networks Neural networks arise from attempts to model human/animal brains Many models, many claims of biological plausibility We will
CSE446: Neural Networks Spring Many slides are adapted from Carlos Guestrin and Luke Zettlemoyer
CSE446: Neural Networks Spring 2017 Many slides are adapted from Carlos Guestrin and Luke Zettlemoyer Human Neurons Switching time ~ 0.001 second Number of neurons 10 10 Connections per neuron 10 4-5 Scene
SGD and Deep Learning
SGD and Deep Learning Subgradients Lets make the gradient cheating more formal. Recall that the gradient is the slope of the tangent. f(w 1 )+rf(w 1 ) (w w 1 ) Non differentiable case? w 1 Subgradients
Jakub Hajic Artificial Intelligence Seminar I
Jakub Hajic Artificial Intelligence Seminar I. 11. 11. 2014 Outline Key concepts Deep Belief Networks Convolutional Neural Networks A couple of questions Convolution Perceptron Feedforward Neural Network
Introduction Neural Networks - Architecture Network Training Small Example - ZIP Codes Summary. Neural Networks - I. Henrik I Christensen
Neural Networks - I Henrik I Christensen Robotics & Intelligent Machines @ GT Georgia Institute of Technology, Atlanta, GA 30332-0280 hic@cc.gatech.edu Henrik I Christensen (RIM@GT) Neural Networks 1 /
Introduction to Convolutional Neural Networks (CNNs)
Introduction to Convolutional Neural Networks (CNNs) nojunk@snu.ac.kr http://mipal.snu.ac.kr Department of Transdisciplinary Studies Seoul National University, Korea Jan. 2016 Many slides are from Fei-Fei
Introduction to Natural Computation. Lecture 9. Multilayer Perceptrons and Backpropagation. Peter Lewis
Introduction to Natural Computation Lecture 9 Multilayer Perceptrons and Backpropagation Peter Lewis 1 / 25 Overview of the Lecture Why multilayer perceptrons? Some applications of multilayer perceptrons.
Multilayer Neural Networks. (sometimes called Multilayer Perceptrons or MLPs)
Multilayer Neural Networks (sometimes called Multilayer Perceptrons or MLPs) Linear separability Hyperplane In 2D: w 1 x 1 + w 2 x 2 + w 0 = 0 Feature 1 x 2 = w 1 w 2 x 1 w 0 w 2 Feature 2 A perceptron
y(x n, w) t n 2. (1)
Network training: Training a neural network involves determining the weight parameter vector w that minimizes a cost function. Given a training set comprising a set of input vector {x n }, n = 1,...N,
Neural Networks. Yan Shao Department of Linguistics and Philology, Uppsala University 7 December 2016
Neural Networks Yan Shao Department of Linguistics and Philology, Uppsala University 7 December 2016 Outline Part 1 Introduction Feedforward Neural Networks Stochastic Gradient Descent Computational Graph
Convolutional Neural Network Architecture
Convolutional Neural Network Architecture Zhisheng Zhong Feburary 2nd, 2018 Zhisheng Zhong Convolutional Neural Network Architecture Feburary 2nd, 2018 1 / 55 Outline 1 Introduction of Convolution Motivation
Neural Architectures for Image, Language, and Speech Processing
Neural Architectures for Image, Language, and Speech Processing Karl Stratos June 26, 2018 1 / 31 Overview Feedforward Networks Need for Specialized Architectures Convolutional Neural Networks (CNNs) Recurrent
Introduction to Machine Learning (67577)
Introduction to Machine Learning (67577) Shai Shalev-Shwartz School of CS and Engineering, The Hebrew University of Jerusalem Deep Learning Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks
Statistical Machine Learning
Statistical Machine Learning Lecture 9 Numerical optimization and deep learning Niklas Wahlström Division of Systems and Control Department of Information Technology Uppsala University niklas.wahlstrom@it.uu.se
Homework 3 COMS 4705 Fall 2017 Prof. Kathleen McKeown
Homework 3 COMS 4705 Fall 017 Prof. Kathleen McKeown The assignment consists of a programming part and a written part. For the programming part, make sure you have set up the development environment as
Machine Learning Basics
Security and Fairness of Deep Learning Machine Learning Basics Anupam Datta CMU Spring 2019 Image Classification Image Classification Image classification pipeline Input: A training set of N images, each
1 Introduction 2. 4 Q-Learning The Q-value The Temporal Difference The whole Q-Learning process... 5
Table of contents 1 Introduction 2 2 Markov Decision Processes 2 3 Future Cumulative Reward 3 4 Q-Learning 4 4.1 The Q-value.............................................. 4 4.2 The Temporal Difference.......................................
Revision: Neural Network
Revision: Neural Network Exercise 1 Tell whether each of the following statements is true or false by checking the appropriate box. Statement True False a) A perceptron is guaranteed to perfectly learn
What Do Neural Networks Do? MLP Lecture 3 Multi-layer networks 1
What Do Neural Networks Do? MLP Lecture 3 Multi-layer networks 1 Multi-layer networks Steve Renals Machine Learning Practical MLP Lecture 3 7 October 2015 MLP Lecture 3 Multi-layer networks 2 What Do Single
COMP 551 Applied Machine Learning Lecture 14: Neural Networks
COMP 551 Applied Machine Learning Lecture 14: Neural Networks Instructor: Ryan Lowe (ryan.lowe@mail.mcgill.ca) Slides mostly by: Class web page: www.cs.mcgill.ca/~hvanho2/comp551 Unless otherwise noted,
Artificial Neural Networks
Artificial Neural Networks Oliver Schulte - CMPT 310 Neural Networks Neural networks arise from attempts to model human/animal brains Many models, many claims of biological plausibility We will focus on
Artificial Neural Networks. Introduction to Computational Neuroscience Tambet Matiisen
Artificial Neural Networks Introduction to Computational Neuroscience Tambet Matiisen 2.04.2018 Artificial neural network NB! Inspired by biology, not based on biology! Applications Automatic speech recognition
Introduction to Machine Learning
Introduction to Machine Learning Neural Networks Varun Chandola x x 5 Input Outline Contents February 2, 207 Extending Perceptrons 2 Multi Layered Perceptrons 2 2. Generalizing to Multiple Labels.................
Multilayer Perceptrons and Backpropagation
Multilayer Perceptrons and Backpropagation Informatics 1 CG: Lecture 7 Chris Lucas School of Informatics University of Edinburgh January 31, 2017 (Slides adapted from Mirella Lapata s.) 1 / 33 Reading:
Neural Networks: Backpropagation
Neural Networks: Backpropagation Seung-Hoon Na 1 1 Department of Computer Science Chonbuk National University 2018.10.25 eung-hoon Na (Chonbuk National University) Neural Networks: Backpropagation 2018.10.25
How to do backpropagation in a brain
How to do backpropagation in a brain Geoffrey Hinton Canadian Institute for Advanced Research & University of Toronto & Google Inc. Prelude I will start with three slides explaining a popular type of deep
Machine Learning. Neural Networks. (slides from Domingos, Pardo, others)
Machine Learning Neural Networks (slides from Domingos, Pardo, others) For this week, Reading Chapter 4: Neural Networks (Mitchell, 1997) See Canvas For subsequent weeks: Scaling Learning Algorithms toward
Lab 5: 16 th April Exercises on Neural Networks
Lab 5: 16 th April 01 Exercises on Neural Networks 1. What are the values of weights w 0, w 1, and w for the perceptron whose decision surface is illustrated in the figure? Assume the surface crosses the
Neural Networks and Deep Learning
Neural Networks and Deep Learning Professor Ameet Talwalkar November 12, 2015 Professor Ameet Talwalkar Neural Networks and Deep Learning November 12, 2015 1 / 16 Outline 1 Review of last lecture AdaBoost
Artificial Neural Networks. MGS Lecture 2
Artificial Neural Networks MGS 2018 - Lecture 2 OVERVIEW Biological Neural Networks Cell Topology: Input, Output, and Hidden Layers Functional description Cost functions Training ANNs Back-Propagation
4. Multilayer Perceptrons
4. Multilayer Perceptrons This is a supervised error-correction learning algorithm. 1 4.1 Introduction A multilayer feedforward network consists of an input layer, one or more hidden layers, and an output
Let your machine do the learning
Let your machine do the learning Neural Networks Daniel Hugo Cámpora Pérez Universidad de Sevilla CERN Inverted CERN School of Computing, 6th - 8th March, 2017 Daniel Hugo Cámpora Pérez Let your machine
CS 6501: Deep Learning for Computer Graphics. Basics of Neural Networks. Connelly Barnes
CS 6501: Deep Learning for Computer Graphics Basics of Neural Networks Connelly Barnes Overview Simple neural networks Perceptron Feedforward neural networks Multilayer perceptron and properties Autoencoders
Neural Networks. Single-layer neural network. CSE 446: Machine Learning Emily Fox University of Washington March 10, /9/17
3/9/7 Neural Networks Emily Fox University of Washington March 0, 207 Slides adapted from Ali Farhadi (via Carlos Guestrin and Luke Zettlemoyer) Single-layer neural network 3/9/7 Perceptron as a neural
Backpropagation and Neural Networks part 1. Lecture 4-1
Lecture 4: Backpropagation and Neural Networks part 1 Lecture 4-1 Administrative A1 is due Jan 20 (Wednesday). ~150 hours left Warning: Jan 18 (Monday) is Holiday (no class/office hours) Also note: Lectures
Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials Philipp Krähenbühl and Vladlen Koltun Stanford University Presenter: Yuan-Ting Hu 1 Conditional Random Field (CRF) E x I = φ u
CSCI 315: Artificial Intelligence through Deep Learning
CSCI 35: Artificial Intelligence through Deep Learning W&L Fall Term 27 Prof. Levy Convolutional Networks http://wernerstudio.typepad.com/.a/6ad83549adb53ef53629ccf97c-5wi Convolution: Convolution is
SPSS, University of Texas at Arlington. Topics in Machine Learning-EE 5359 Neural Networks
Topics in Machine Learning-EE 5359 Neural Networks 1 The Perceptron Output: A perceptron is a function that maps D-dimensional vectors to real numbers. For notational convenience, we add a zero-th dimension
Machine Learning. Neural Networks
Machine Learning Neural Networks Bryan Pardo, Northwestern University, Machine Learning EECS 349 Fall 2007 Biological Analogy Bryan Pardo, Northwestern University, Machine Learning EECS 349 Fall 2007 THE
Reading Group on Deep Learning Session 1
Reading Group on Deep Learning Session 1 Stephane Lathuiliere & Pablo Mesejo 2 June 2016 1/31 Contents Introduction to Artificial Neural Networks to understand, and to be able to efficiently use, the popular
Neural networks and optimization
Neural networks and optimization Nicolas Le Roux Criteo 18/05/15 Nicolas Le Roux (Criteo) Neural networks and optimization 18/05/15 1 / 85 1 Introduction 2 Deep networks 3 Optimization 4 Convolutional
Online Videos FERPA. Sign waiver or sit on the sides or in the back. Off camera question time before and after lecture. Questions?
Online Videos FERPA Sign waiver or sit on the sides or in the back Off camera question time before and after lecture Questions? Lecture 1, Slide 1 CS224d Deep NLP Lecture 4: Word Window Classification
Unit III. A Survey of Neural Network Model
Unit III A Survey of Neural Network Model 1 Single Layer Perceptron Perceptron the first adaptive network architecture was invented by Frank Rosenblatt in 1957. It can be used for the classification of
Nonlinear Classification
Nonlinear Classification INFO-4604, Applied Machine Learning University of Colorado Boulder October 5-10, 2017 Prof. Michael Paul Linear Classification Most classifiers we ve seen use linear functions
Spatial Transformation
Spatial Transformation Presented by Liqun Chen June 30, 2017 1 Overview 2 Spatial Transformer Networks 3 STN experiments 4 Recurrent Models of Visual Attention (RAM) 5 Recurrent Models of Visual Attention
Neural Networks. David Rosenberg. July 26, New York University. David Rosenberg (New York University) DS-GA 1003 July 26, / 35
Neural Networks David Rosenberg New York University July 26, 2017 David Rosenberg (New York University) DS-GA 1003 July 26, 2017 1 / 35 Neural Networks Overview Objectives What are neural networks? How
Introduction to Neural Networks
Introduction to Neural Networks Steve Renals Automatic Speech Recognition ASR Lecture 10 24 February 2014 ASR Lecture 10 Introduction to Neural Networks 1 Neural networks for speech recognition Introduction
Speech and Language Processing
Speech and Language Processing Lecture 5 Neural network based acoustic and language models Information and Communications Engineering Course Takahiro Shinoaki 08//6 Lecture Plan (Shinoaki s part) I gives
Lecture 5: Logistic Regression. Neural Networks
Lecture 5: Logistic Regression. Neural Networks Logistic regression Comparison with generative models Feed-forward neural networks Backpropagation Tricks for training neural networks COMP-652, Lecture
From perceptrons to word embeddings. Simon Šuster University of Groningen
From perceptrons to word embeddings Simon Šuster University of Groningen Outline A basic computational unit Weighting some input to produce an output: classification Perceptron Classify tweets Written
Machine Learning. Neural Networks. (slides from Domingos, Pardo, others)
Machine Learning Neural Networks (slides from Domingos, Pardo, others) For this week, Reading Chapter 4: Neural Networks (Mitchell, 1997) See Canvas For subsequent weeks: Scaling Learning Algorithms toward
Machine Learning and Data Mining. Multi-layer Perceptrons & Neural Networks: Basics. Prof. Alexander Ihler
+ Machine Learning and Data Mining Multi-layer Perceptrons & Neural Networks: Basics Prof. Alexander Ihler Linear Classifiers (Perceptrons) Linear Classifiers a linear classifier is a mapping which partitions
Logistic Regression. COMP 527 Danushka Bollegala
Logistic Regression COMP 527 Danushka Bollegala Binary Classification Given an instance x we must classify it to either positive (1) or negative (0) class We can use {1,-1} instead of {1,0} but we will
Neural Networks with Applications to Vision and Language. Feedforward Networks. Marco Kuhlmann
Neural Networks with Applications to Vision and Language Feedforward Networks Marco Kuhlmann Feedforward networks Linear separability x 2 x 2 0 1 0 1 0 0 x 1 1 0 x 1 linearly separable not linearly separable
Feature Design. Feature Design. Feature Design. & Deep Learning
Artificial Intelligence and its applications Lecture 9 & Deep Learning Professor Daniel Yeung danyeung@ieee.org Dr. Patrick Chan patrickchan@ieee.org South China University of Technology, China Appropriately
Recurrent Neural Networks
Recurrent Neural Networks Datamining Seminar Kaspar Märtens Karl-Oskar Masing Today's Topics Modeling sequences: a brief overview Training RNNs with back propagation A toy example of training an RNN Why
Sajid Anwar, Kyuyeon Hwang and Wonyong Sung
Sajid Anwar, Kyuyeon Hwang and Wonyong Sung Department of Electrical and Computer Engineering Seoul National University Seoul, 08826 Korea Email: sajid@dsp.snu.ac.kr, khwang@dsp.snu.ac.kr, wysung@snu.ac.kr
Mitosis Detection in Breast Cancer Histology Images with Multi Column Deep Neural Networks
Mitosis Detection in Breast Cancer Histology Images with Multi Column Deep Neural Networks IDSIA, Lugano, Switzerland dan.ciresan@gmail.com Dan C. Cireşan and Alessandro Giusti DNN for Visual Pattern Recognition
Intro to Neural Networks and Deep Learning
Intro to Neural Networks and Deep Learning Jack Lanchantin Dr. Yanjun Qi UVA CS 6316 1 Neurons 1-Layer Neural Network Multi-layer Neural Network Loss Functions Backpropagation Nonlinearity Functions NNs
Binary Convolutional Neural Network on RRAM
Binary Convolutional Neural Network on RRAM Tianqi Tang, Lixue Xia, Boxun Li, Yu Wang, Huazhong Yang Dept. of E.E, Tsinghua National Laboratory for Information Science and Technology (TNList) Tsinghua
Comments. Assignment 3 code released. Thought questions 3 due this week. Mini-project: hopefully you have started. implement classification algorithms
Neural networks Comments Assignment 3 code released implement classification algorithms use kernels for census dataset Thought questions 3 due this week Mini-project: hopefully you have started 2 Example:
Statistical NLP for the Web
Statistical NLP for the Web Neural Networks, Deep Belief Networks Sameer Maskey Week 8, October 24, 2012 *some slides from Andrew Rosenberg Announcements Please ask HW2 related questions in courseworks