C4 Phenomenological Modeling - Regression & Neural Networks : Computational Modeling and Simulation Instructor: Linwei Wang
|
|
- Malcolm Burns
- 6 years ago
- Views:
Transcription
1 C4 Phenomenological Modeling - Regression & Neural Networks : Computational Modeling and Simulation Instructor: Linwei Wang
2 Recall.. The simple, multiple linear regression function ŷ(x) = a 0 + a 1 x 1 + a 2 x a n x n Which can be viewed as a black-box in the right: Each of the input node values is multiplied with a corresponding weight, the results are added up, and the output value is obtained as this sum plus a constant (the so-called bias) This is exactly what one neuron does in the ANN!
3 Building Blocks of ANN Our crude way to simulate the brain electronically Multiple inputs Weights can be negative to represent excitory or inhibitory influences Output: activation
4 Artificial Neural Network Mathematically, an artificial neuron is modeled as: d(x) = f (w T x + w 0 ) where f is a non-linear function (transfer/activation function), e.g. Threshold Sigmoid " $ f (y) = 0 y < 0 # % $ 1 y! 0 1 f (y) = 1+ e &cy f (y) = 1! a tan(y)+ 1 2 Mul:ple linear regression + a nonlinear ac:va:on func:on
5 Feed-Forward Neural Network Put all the neurons into a network Arbitrary number of layers It is common to use two layers Arbitrary number of ANs in each layer Common ANNs have 3 layers Input layer: x Hidden layer: z Output layer: y z i = f h (w i T x + w i,0 ) y = f o (v T z + v 0 ) Number of parameters to tune h(n +1)+ m(h +1) Each layer acts in the same way but with different coefficients and/or nonlinear func:ons
6 Feed-Forward Neural Network Generalize: skip-layer connections Let s still look at 3 layers Input layer: x Hidden layer: z Output layer: y z i = f h (w i T x + w i,0 ) Number of parameter to tune h(n +1)+ m(h +1)+ mn y = f o (v T z + v 0 + w o T x) The single- hidden- layer feedforward neural network can approximate any con:nuous func:on by increasing the size of the hidden layer
7 The Learning Process Each neural network possesses knowledge contained in the values of the connected weights Modifying the knowledge stored in the network as a function of experiences implies a learning rule for changing the values of the weights Minimization of RSQ Learning as a gradient descent Back-propagation algorithm m min{ 1! ( y 2 i=1 i " ŷ i ) 2 )
8 Back-Propagation Algorithm To adjust the weights for each unit such that the error between the desired output and the actual output is reduced (minimizing RSQ) Learning using the method of gradient descent To compute the gradient of the error function, i.e., error derivative of the weights EW (how the error changes as each weight is increased or decreased slightly) Must guarantee the continuity and differentiability of the error function Activation function: e.g. sigmoid function f (x) = 1 df (x) = 1+ e!cx dx e!x (1+ e!cx ) 2 = f (x)(1! f (x))
9 Back-Propagation Algorithm Activation function: e.g. sigmoid function Guarantee the continuity and differentiability of the error function Valued between 0 and 1 Local minima could occur 1 f (x) = n 1+ exp( w i x i + w 0 )! i=1
10 Back-Propagation Algorithm Find a local minimum of the error function E = 1 m! ( y 2 i=1 i " ŷ i ) 2 ANN is initialized with randomly chosen weights The gradient of the error function, EW, is computed and used to correct the initial weights EW is computed recursively!e = ( "E "w 1, "E "w 2,..., "E "w l )!w i = "! #E #w i i =1,...,l Assume l weights in the network r: learning constant, defines the step length of each itera:on in the nega:ve gradient direc:on
11 Back-Propagation Algorithm Now let s forget about training sets and learning Our objective is to find a method for efficiently calculating the gradient of network function according to the weights of the network Because our network = a complex chain of function compositions (addition, weighted edge, nonlinear activation), we expect the chain rules of calculus to play a major role in finding the gradient Let s start with a 1D network
12 B-Diagram Feed-forward step: Info comes from the left and each unit evaluates function f in its right side and the derivation f in left side Both results are stored in the unit, only that on the right side is transmitted to the units connected to the right Backpropagation step Running the whole network backwards, using the stored results Deriva:ve of the func:on Single compu:ng unit func:on Separa:on into addi:on and ac:va:on unit
13 Three Basic Cases Function composition Forward: Backward: The input from the right of the network is the constant 1 Incoming info is multiplied by the value stored in its left side The results (traversing value) is the derivative of the function composition Any sequence of function compositions can be evaluated in this way & its derivative obtained in the backpropagation step The network being used backwards with the input 1 At each node the product with the value stored in the left side is computed
14 Three Basic Cases Function addition Forward: Backward: All incoming edges to a unit fan out the traversing value at this node and distribute it to the connected units to the left When two right-to-left paths meet, the computed traversing values are added
15 Three Basic Cases Weighted edges Forward: Backward
16 Steps of the Backpropagation Algorithm Consider a network with a single real input x and network function F, the derivative F (x) is computed in two phases: Feedforward: The input x is fed into the network. The primitive functions at the nodes and their derivatives are evaluated at each node & stored Backpropagation: The constant 1 is fed into the output unit and the network is fun backwards. Incoming info to a node is added and the result is multiplied by the values stored in the left part of the unit. The result is transmitted to the left of the unit. The result collected at the input unit is the F (x) We can prove that it works in arbitrary feed-forward networks with differentiable activation functions at the nodes
17 Steps of the Backpropagation Algorithm F(x) =!(w 1 F 1 (x)+ w 2 F 2 (x) w m F m (x)) F '(x) =! '(s)(w 1 F ' 1 (x)+ w 2 F ' 2 (x) w m F ' m (x))
18 Generalization to More Inputs The feed-forward step remains unchanged & all left side slots of the units are filled as usual In the backpropagation we can identify two subnetworks
19 Learning with Backpropagation The feed-forward step is computed in the usual way, but we also store the output of each unit in its right side We perform the backpropagation in the network If we fix our attention on one of the weights, say w ij whose associated edge points from the i-th to the j-th node in the network The weight can be treated as input channel into the subnetwork made of all paths starting at w ij and ending in the single output unit of the network The info fed into the subnetwork in the feed-forward step was o i w ij (o i the stored output of unit i) The backpropagation computes the gradient of error E with respect to this input!e!e Usual result in backpropaga:on at = o i one node with regard to one input!w ij!o i w ij
20 Learning with Backpropagation The backpropagation is performed in the usual way. All subnetworks defined by each weight of the network can be handled simultaneously, but we store additionally at each node i The output o i of the node in the feed-forward step The cumulative result of the backward computation up to this node (backpropagated error δ j )!E!w ij = o i! j Once all partial derivatives are computed, we can perform gradient descent by adding to each weight:!w ij = "!o i " j
21 Layered Networks Notation: n input, k hidden, m output Weights and matrix W 1,W 2 The excitation net of the j-th hidden units n+1 (1) net j =! w ij ô i i=1 The outputs of this unit = s( n+1 w (1) j! ij ô i ) o (1) i=1 In matrix form o (1) = s(ôw 1 ) o (2) = s(ô (1) W 2 )
22 Layered Networks Let s consider a single input-output pair (o,t), i.e., 1 training set Backpropagation Feedforward computation Backpropagation to the output layer Backpropagation to the hidden layer Weights updates Stops when the value of the error functions is sufficiently sall Extended network for compu:ng error
23 Layered Networks Feedward computation The vector o is presented to the network, the vectors o (1) and o (2) are computed and stored. The derivatives of the activation functions are also stored at each unit Backpropagation to the output layer!e Interested in!w (2) ij Extended network for compu:ng error
24 Layered Networks Backpropagation to the output layer Interested in!e /!w (2) ij Bakpropagated error! j (2) = o j (2) (1! o j (2) )(o j (2)! t j ) Partial derivative!e /!w (2) (1) ij = o i! (2) j = [o (2) j (1" o (2) j )(o (2) (1) j " t j )]o i
25 Layered Networks Backpropagation to the hidden layer Interested in!e /!w (1) ij Bakpropagated error! (1) j = o (1) j (1! o (1) m j ) w (2) (2) " jq! q q=1 Partial derivative!e /!w (1) (1) ij = o i! j
26 Layered Networks Weights update Hidden-output layer!w (2) = "!o (1) ij i " (2) j, i =1,..., k +1; j =1,...m Input-hidden layer o n+1 = o (1) k+1 =1!w (1) ij = "!o i " j (1), i =1,..., n +1; j =1,...k Make the corrections to the weight only after the backpropagated error has been computed for all units in the network!!!! Otherwise the corrections become interwined with the backpropagation, and the computed corrections do not correspond to the negative gradient direction
27 More than One Training Set If we have p datasets Batch / offline updates! 1 w (1) ij,! 2 w (1) (1) ij,...! p w ij The necessary updates:!w (1) ij =! 1 w (1) ij +! 2 w (1) (1) ij ! p w ij Online / sequential updates The corrections do not exactly follow the negative gradient direction If the training sets are selected randomly, the search direction oscillates around the exact gradient direction and, on average, the algorithm implements a form of descent in the error function Adding some noise to the gradient function can help to avoid falling into shallow local minima It is very expensive to compute the exact gradient direction when the training set is large
28 Backpropagation in Matrix Forms Input-output: o (2) = s(ô (1) W 2 ) o (1) = s(ô W 1 ) The derivatives (stored in the feed-forward step) o 1 (2) (1! o 1 (2) ) o 1 (1) (1! o 1 (1) ) D 2 = ( 0 o (2) 2 (1! o (2) 2 )... 0!! "! o (2) m (1! o (2) m ) ) D 2 = ( 0 o (1) 2 (1! o (1) 2 )... 0!! "! o (1) k (1! o (1) k ) ) The stored derivatives of the quadratic error " o (2) 1! t 1 % $ ' $ o (2) e = 2! t 2 ' $! ' $ # o (2) ' m! t m &
29 Backpropagation in Matrix Forms The m-dimensional vector of the backpropagated error up to the output units! (2) = D 2 e The k-dimensional vectors of the backpropagated error up to the hidden layer! (1) = D 1 W 2! (2) The correction for the two weight matrices!w T 2 = "!" (2) ô (1),!W T 1 = "!" (1) ô We can generalize this for l-layers! (l) = D l e! (i) = D i W i+1! (i+1) i =1,..l!1 Or! (i) = D i W i+1...w l!1 D l!1 W l D l e
30 Back-Propagation Summary First, compute EA: rate at which the error changes as the activity level of a unit is changed Output layer: the difference between the actual and desired outputs Hidden layer: Identifying all the weights between that hidden unit and the outputs it connected with Multiply those weights by the EAs of those output units and add the products Other layers: Similar fashion, calculated from layer to layer in a direction opposite to the way activities propagate through the network (hence the name back-propagation) Second, EW for each connection of the unit is the product of the EA and the activity through the incoming connection
31 More General ANN Feed-forward ANN Signals travel one way only: from input to output Feedback ANN Signals travel in both directions through loops in the network Very powerful and can get extremely complicated Automatic detection of nonlinearities: ANN describes the nonlinear dependency of the response variable on the independent variables without a previous explicit specification of this nonlinear dependency
32 Generalization and Overfitting Back to the investment example: 3- layer ANN h = 3 h = 6 Which model is beqer?
33 Generalization and Overfitting Generalization: Suppose two mathematical models (S,Q,M) and (S,Q,M * ) have been setup using a training dataset D train. Then (S,Q,M) is said to generalize better than (S,Q,M * ) on a test dataset D test with respect to some error criterion E, if (S,Q,M) produces a smaller value of E on D test compared to (S,Q,M * ) Not sufficient to look at a model s performance only on the dataset used to construct the model, if one wants to achieve good predictive capabilities Better predictions are obtained from models which describe the essential tendency of the data instead of following random oscillations
34 Generalization and Overfitting Overfitting: A mathematical model (S,Q,M) is said to overfit a training dataset D train with respect to an error criterion E and a test dataset D test, if another model (S,Q,M * ) with a larger error on D train generalizes better to D test Regularization methods can be used to reduce overfitting, using modified fitting criteria that penalize the roughness of the ANN Weight decay Roughness is associated with large values of the weight parameters The sum of squares of the network is included in the fitting criterion
Medical Image Recognition Linwei Wang
Medical Image Recognition 4005-759 Linwei Wang Content Classification methods in medical images Statistical Classification: Bayes Method Distance-Based: Nearest Neighbor Method Region-Based: Support Vector
More informationy(x n, w) t n 2. (1)
Network training: Training a neural network involves determining the weight parameter vector w that minimizes a cost function. Given a training set comprising a set of input vector {x n }, n = 1,...N,
More information4. Multilayer Perceptrons
4. Multilayer Perceptrons This is a supervised error-correction learning algorithm. 1 4.1 Introduction A multilayer feedforward network consists of an input layer, one or more hidden layers, and an output
More informationData Mining Part 5. Prediction
Data Mining Part 5. Prediction 5.5. Spring 2010 Instructor: Dr. Masoud Yaghini Outline How the Brain Works Artificial Neural Networks Simple Computing Elements Feed-Forward Networks Perceptrons (Single-layer,
More informationNeural Networks and the Back-propagation Algorithm
Neural Networks and the Back-propagation Algorithm Francisco S. Melo In these notes, we provide a brief overview of the main concepts concerning neural networks and the back-propagation algorithm. We closely
More informationMultilayer Perceptron
Outline Hong Chang Institute of Computing Technology, Chinese Academy of Sciences Machine Learning Methods (Fall 2012) Outline Outline I 1 Introduction 2 Single Perceptron 3 Boolean Function Learning 4
More informationNeural Networks and Deep Learning
Neural Networks and Deep Learning Professor Ameet Talwalkar November 12, 2015 Professor Ameet Talwalkar Neural Networks and Deep Learning November 12, 2015 1 / 16 Outline 1 Review of last lecture AdaBoost
More informationCS 6501: Deep Learning for Computer Graphics. Basics of Neural Networks. Connelly Barnes
CS 6501: Deep Learning for Computer Graphics Basics of Neural Networks Connelly Barnes Overview Simple neural networks Perceptron Feedforward neural networks Multilayer perceptron and properties Autoencoders
More informationFeedforward Neural Nets and Backpropagation
Feedforward Neural Nets and Backpropagation Julie Nutini University of British Columbia MLRG September 28 th, 2016 1 / 23 Supervised Learning Roadmap Supervised Learning: Assume that we are given the features
More informationCOMP 551 Applied Machine Learning Lecture 14: Neural Networks
COMP 551 Applied Machine Learning Lecture 14: Neural Networks Instructor: Ryan Lowe (ryan.lowe@mail.mcgill.ca) Slides mostly by: Class web page: www.cs.mcgill.ca/~hvanho2/comp551 Unless otherwise noted,
More informationAI Programming CS F-20 Neural Networks
AI Programming CS662-2008F-20 Neural Networks David Galles Department of Computer Science University of San Francisco 20-0: Symbolic AI Most of this class has been focused on Symbolic AI Focus or symbols
More informationArtificial Intelligence
Artificial Intelligence Jeff Clune Assistant Professor Evolving Artificial Intelligence Laboratory Announcements Be making progress on your projects! Three Types of Learning Unsupervised Supervised Reinforcement
More informationNeural Networks, Computation Graphs. CMSC 470 Marine Carpuat
Neural Networks, Computation Graphs CMSC 470 Marine Carpuat Binary Classification with a Multi-layer Perceptron φ A = 1 φ site = 1 φ located = 1 φ Maizuru = 1 φ, = 2 φ in = 1 φ Kyoto = 1 φ priest = 0 φ
More informationMachine Learning and Data Mining. Multi-layer Perceptrons & Neural Networks: Basics. Prof. Alexander Ihler
+ Machine Learning and Data Mining Multi-layer Perceptrons & Neural Networks: Basics Prof. Alexander Ihler Linear Classifiers (Perceptrons) Linear Classifiers a linear classifier is a mapping which partitions
More informationSPSS, University of Texas at Arlington. Topics in Machine Learning-EE 5359 Neural Networks
Topics in Machine Learning-EE 5359 Neural Networks 1 The Perceptron Output: A perceptron is a function that maps D-dimensional vectors to real numbers. For notational convenience, we add a zero-th dimension
More informationIntroduction to Natural Computation. Lecture 9. Multilayer Perceptrons and Backpropagation. Peter Lewis
Introduction to Natural Computation Lecture 9 Multilayer Perceptrons and Backpropagation Peter Lewis 1 / 25 Overview of the Lecture Why multilayer perceptrons? Some applications of multilayer perceptrons.
More informationComputational statistics
Computational statistics Lecture 3: Neural networks Thierry Denœux 5 March, 2016 Neural networks A class of learning methods that was developed separately in different fields statistics and artificial
More informationLecture 4: Feed Forward Neural Networks
Lecture 4: Feed Forward Neural Networks Dr. Roman V Belavkin Middlesex University BIS4435 Biological neurons and the brain A Model of A Single Neuron Neurons as data-driven models Neural Networks Training
More informationArtifical Neural Networks
Neural Networks Artifical Neural Networks Neural Networks Biological Neural Networks.................................. Artificial Neural Networks................................... 3 ANN Structure...........................................
More informationCOGS Q250 Fall Homework 7: Learning in Neural Networks Due: 9:00am, Friday 2nd November.
COGS Q250 Fall 2012 Homework 7: Learning in Neural Networks Due: 9:00am, Friday 2nd November. For the first two questions of the homework you will need to understand the learning algorithm using the delta
More informationCSC242: Intro to AI. Lecture 21
CSC242: Intro to AI Lecture 21 Administrivia Project 4 (homeworks 18 & 19) due Mon Apr 16 11:59PM Posters Apr 24 and 26 You need an idea! You need to present it nicely on 2-wide by 4-high landscape pages
More informationNeural Nets Supervised learning
6.034 Artificial Intelligence Big idea: Learning as acquiring a function on feature vectors Background Nearest Neighbors Identification Trees Neural Nets Neural Nets Supervised learning y s(z) w w 0 w
More informationCSC 411 Lecture 10: Neural Networks
CSC 411 Lecture 10: Neural Networks Roger Grosse, Amir-massoud Farahmand, and Juan Carrasquilla University of Toronto UofT CSC 411: 10-Neural Networks 1 / 35 Inspiration: The Brain Our brain has 10 11
More informationNeed for Deep Networks Perceptron. Can only model linear functions. Kernel Machines. Non-linearity provided by kernels
Need for Deep Networks Perceptron Can only model linear functions Kernel Machines Non-linearity provided by kernels Need to design appropriate kernels (possibly selecting from a set, i.e. kernel learning)
More informationEPL442: Computational
EPL442: Computational Learning Systems Lab 2 Vassilis Vassiliades Department of Computer Science University of Cyprus Outline Artificial Neuron Feedforward Neural Network Back-propagation Algorithm Notes
More informationNeural Networks. CSE 6363 Machine Learning Vassilis Athitsos Computer Science and Engineering Department University of Texas at Arlington
Neural Networks CSE 6363 Machine Learning Vassilis Athitsos Computer Science and Engineering Department University of Texas at Arlington 1 Perceptrons x 0 = 1 x 1 x 2 z = h w T x Output: z x D A perceptron
More informationNONLINEAR CLASSIFICATION AND REGRESSION. J. Elder CSE 4404/5327 Introduction to Machine Learning and Pattern Recognition
NONLINEAR CLASSIFICATION AND REGRESSION Nonlinear Classification and Regression: Outline 2 Multi-Layer Perceptrons The Back-Propagation Learning Algorithm Generalized Linear Models Radial Basis Function
More informationMachine Learning
Machine Learning 10-315 Maria Florina Balcan Machine Learning Department Carnegie Mellon University 03/29/2019 Today: Artificial neural networks Backpropagation Reading: Mitchell: Chapter 4 Bishop: Chapter
More informationSupervised Learning in Neural Networks
The Norwegian University of Science and Technology (NTNU Trondheim, Norway keithd@idi.ntnu.no March 7, 2011 Supervised Learning Constant feedback from an instructor, indicating not only right/wrong, but
More informationARTIFICIAL NEURAL NETWORK PART I HANIEH BORHANAZAD
ARTIFICIAL NEURAL NETWORK PART I HANIEH BORHANAZAD WHAT IS A NEURAL NETWORK? The simplest definition of a neural network, more properly referred to as an 'artificial' neural network (ANN), is provided
More informationMultilayer Perceptrons and Backpropagation
Multilayer Perceptrons and Backpropagation Informatics 1 CG: Lecture 7 Chris Lucas School of Informatics University of Edinburgh January 31, 2017 (Slides adapted from Mirella Lapata s.) 1 / 33 Reading:
More informationArtificial Neural Networks
Artificial Neural Networks Threshold units Gradient descent Multilayer networks Backpropagation Hidden layer representations Example: Face Recognition Advanced topics 1 Connectionist Models Consider humans:
More informationMachine Learning for Large-Scale Data Analysis and Decision Making A. Neural Networks Week #6
Machine Learning for Large-Scale Data Analysis and Decision Making 80-629-17A Neural Networks Week #6 Today Neural Networks A. Modeling B. Fitting C. Deep neural networks Today s material is (adapted)
More information22c145-Fall 01: Neural Networks. Neural Networks. Readings: Chapter 19 of Russell & Norvig. Cesare Tinelli 1
Neural Networks Readings: Chapter 19 of Russell & Norvig. Cesare Tinelli 1 Brains as Computational Devices Brains advantages with respect to digital computers: Massively parallel Fault-tolerant Reliable
More informationIntroduction to Neural Networks
Introduction to Neural Networks What are (Artificial) Neural Networks? Models of the brain and nervous system Highly parallel Process information much more like the brain than a serial computer Learning
More informationArtificial Neural Networks. MGS Lecture 2
Artificial Neural Networks MGS 2018 - Lecture 2 OVERVIEW Biological Neural Networks Cell Topology: Input, Output, and Hidden Layers Functional description Cost functions Training ANNs Back-Propagation
More informationIntroduction to Neural Networks
CUONG TUAN NGUYEN SEIJI HOTTA MASAKI NAKAGAWA Tokyo University of Agriculture and Technology Copyright by Nguyen, Hotta and Nakagawa 1 Pattern classification Which category of an input? Example: Character
More informationVasil Khalidov & Miles Hansard. C.M. Bishop s PRML: Chapter 5; Neural Networks
C.M. Bishop s PRML: Chapter 5; Neural Networks Introduction The aim is, as before, to find useful decompositions of the target variable; t(x) = y(x, w) + ɛ(x) (3.7) t(x n ) and x n are the observations,
More informationMachine Learning
Machine Learning 10-601 Maria Florina Balcan Machine Learning Department Carnegie Mellon University 02/10/2016 Today: Artificial neural networks Backpropagation Reading: Mitchell: Chapter 4 Bishop: Chapter
More informationMultilayer Neural Networks
Multilayer Neural Networks Introduction Goal: Classify objects by learning nonlinearity There are many problems for which linear discriminants are insufficient for minimum error In previous methods, the
More informationMultilayer Neural Networks
Multilayer Neural Networks Multilayer Neural Networks Discriminant function flexibility NON-Linear But with sets of linear parameters at each layer Provably general function approximators for sufficient
More informationSerious limitations of (single-layer) perceptrons: Cannot learn non-linearly separable tasks. Cannot approximate (learn) non-linear functions
BACK-PROPAGATION NETWORKS Serious limitations of (single-layer) perceptrons: Cannot learn non-linearly separable tasks Cannot approximate (learn) non-linear functions Difficult (if not impossible) to design
More informationArtificial Neural Networks. Edward Gatt
Artificial Neural Networks Edward Gatt What are Neural Networks? Models of the brain and nervous system Highly parallel Process information much more like the brain than a serial computer Learning Very
More informationNeed for Deep Networks Perceptron. Can only model linear functions. Kernel Machines. Non-linearity provided by kernels
Need for Deep Networks Perceptron Can only model linear functions Kernel Machines Non-linearity provided by kernels Need to design appropriate kernels (possibly selecting from a set, i.e. kernel learning)
More information(Feed-Forward) Neural Networks Dr. Hajira Jabeen, Prof. Jens Lehmann
(Feed-Forward) Neural Networks 2016-12-06 Dr. Hajira Jabeen, Prof. Jens Lehmann Outline In the previous lectures we have learned about tensors and factorization methods. RESCAL is a bilinear model for
More informationLecture 5: Logistic Regression. Neural Networks
Lecture 5: Logistic Regression. Neural Networks Logistic regression Comparison with generative models Feed-forward neural networks Backpropagation Tricks for training neural networks COMP-652, Lecture
More informationArtificial Neural Networks
Artificial Neural Networks Oliver Schulte - CMPT 310 Neural Networks Neural networks arise from attempts to model human/animal brains Many models, many claims of biological plausibility We will focus on
More informationIntroduction Neural Networks - Architecture Network Training Small Example - ZIP Codes Summary. Neural Networks - I. Henrik I Christensen
Neural Networks - I Henrik I Christensen Robotics & Intelligent Machines @ GT Georgia Institute of Technology, Atlanta, GA 30332-0280 hic@cc.gatech.edu Henrik I Christensen (RIM@GT) Neural Networks 1 /
More informationNeural networks. Chapter 20. Chapter 20 1
Neural networks Chapter 20 Chapter 20 1 Outline Brains Neural networks Perceptrons Multilayer networks Applications of neural networks Chapter 20 2 Brains 10 11 neurons of > 20 types, 10 14 synapses, 1ms
More informationIntroduction to Machine Learning
Introduction to Machine Learning Neural Networks Varun Chandola x x 5 Input Outline Contents February 2, 207 Extending Perceptrons 2 Multi Layered Perceptrons 2 2. Generalizing to Multiple Labels.................
More informationLearning Neural Networks
Learning Neural Networks Neural Networks can represent complex decision boundaries Variable size. Any boolean function can be represented. Hidden units can be interpreted as new features Deterministic
More informationNeural Networks. Nicholas Ruozzi University of Texas at Dallas
Neural Networks Nicholas Ruozzi University of Texas at Dallas Handwritten Digit Recognition Given a collection of handwritten digits and their corresponding labels, we d like to be able to correctly classify
More informationArtificial Neural Networks
Introduction ANN in Action Final Observations Application: Poverty Detection Artificial Neural Networks Alvaro J. Riascos Villegas University of los Andes and Quantil July 6 2018 Artificial Neural Networks
More informationMachine Learning Basics III
Machine Learning Basics III Benjamin Roth CIS LMU München Benjamin Roth (CIS LMU München) Machine Learning Basics III 1 / 62 Outline 1 Classification Logistic Regression 2 Gradient Based Optimization Gradient
More informationNeural Networks (and Gradient Ascent Again)
Neural Networks (and Gradient Ascent Again) Frank Wood April 27, 2010 Generalized Regression Until now we have focused on linear regression techniques. We generalized linear regression to include nonlinear
More informationPattern Recognition Prof. P. S. Sastry Department of Electronics and Communication Engineering Indian Institute of Science, Bangalore
Pattern Recognition Prof. P. S. Sastry Department of Electronics and Communication Engineering Indian Institute of Science, Bangalore Lecture - 27 Multilayer Feedforward Neural networks with Sigmoidal
More informationNeural Networks. Bishop PRML Ch. 5. Alireza Ghane. Feed-forward Networks Network Training Error Backpropagation Applications
Neural Networks Bishop PRML Ch. 5 Alireza Ghane Neural Networks Alireza Ghane / Greg Mori 1 Neural Networks Neural networks arise from attempts to model human/animal brains Many models, many claims of
More informationNN V: The generalized delta learning rule
NN V: The generalized delta learning rule We now focus on generalizing the delta learning rule for feedforward layered neural networks. The architecture of the two-layer network considered below is shown
More informationFeed-forward Networks Network Training Error Backpropagation Applications. Neural Networks. Oliver Schulte - CMPT 726. Bishop PRML Ch.
Neural Networks Oliver Schulte - CMPT 726 Bishop PRML Ch. 5 Neural Networks Neural networks arise from attempts to model human/animal brains Many models, many claims of biological plausibility We will
More informationIntroduction Biologically Motivated Crude Model Backpropagation
Introduction Biologically Motivated Crude Model Backpropagation 1 McCulloch-Pitts Neurons In 1943 Warren S. McCulloch, a neuroscientist, and Walter Pitts, a logician, published A logical calculus of the
More informationNeural Networks DWML, /25
DWML, 2007 /25 Neural networks: Biological and artificial Consider humans: Neuron switching time 0.00 second Number of neurons 0 0 Connections per neuron 0 4-0 5 Scene recognition time 0. sec 00 inference
More informationStatistical NLP for the Web
Statistical NLP for the Web Neural Networks, Deep Belief Networks Sameer Maskey Week 8, October 24, 2012 *some slides from Andrew Rosenberg Announcements Please ask HW2 related questions in courseworks
More informationBACKPROPAGATION. Neural network training optimization problem. Deriving backpropagation
BACKPROPAGATION Neural network training optimization problem min J(w) w The application of gradient descent to this problem is called backpropagation. Backpropagation is gradient descent applied to J(w)
More informationUnit III. A Survey of Neural Network Model
Unit III A Survey of Neural Network Model 1 Single Layer Perceptron Perceptron the first adaptive network architecture was invented by Frank Rosenblatt in 1957. It can be used for the classification of
More informationArtificial Neural Networks" and Nonparametric Methods" CMPSCI 383 Nov 17, 2011!
Artificial Neural Networks" and Nonparametric Methods" CMPSCI 383 Nov 17, 2011! 1 Todayʼs lecture" How the brain works (!)! Artificial neural networks! Perceptrons! Multilayer feed-forward networks! Error
More informationMark Gales October y (x) x 1. x 2 y (x) Inputs. Outputs. x d. y (x) Second Output layer layer. layer.
University of Cambridge Engineering Part IIB & EIST Part II Paper I0: Advanced Pattern Processing Handouts 4 & 5: Multi-Layer Perceptron: Introduction and Training x y (x) Inputs x 2 y (x) 2 Outputs x
More informationCS:4420 Artificial Intelligence
CS:4420 Artificial Intelligence Spring 2018 Neural Networks Cesare Tinelli The University of Iowa Copyright 2004 18, Cesare Tinelli and Stuart Russell a a These notes were originally developed by Stuart
More information18.6 Regression and Classification with Linear Models
18.6 Regression and Classification with Linear Models 352 The hypothesis space of linear functions of continuous-valued inputs has been used for hundreds of years A univariate linear function (a straight
More informationClassification goals: Make 1 guess about the label (Top-1 error) Make 5 guesses about the label (Top-5 error) No Bounding Box
ImageNet Classification with Deep Convolutional Neural Networks Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton Motivation Classification goals: Make 1 guess about the label (Top-1 error) Make 5 guesses
More informationA thorough derivation of back-propagation for people who really want to understand it by: Mike Gashler, September 2010
A thorough derivation of back-propagation for people who really want to understand it by: Mike Gashler, September 2010 Define the problem: Suppose we have a 5-layer feed-forward neural network. (I intentionally
More informationFrom perceptrons to word embeddings. Simon Šuster University of Groningen
From perceptrons to word embeddings Simon Šuster University of Groningen Outline A basic computational unit Weighting some input to produce an output: classification Perceptron Classify tweets Written
More information8. Lecture Neural Networks
Soft Control (AT 3, RMA) 8. Lecture Neural Networks Learning Process Contents of the 8 th lecture 1. Introduction of Soft Control: Definition and Limitations, Basics of Intelligent" Systems 2. Knowledge
More informationIntro to Neural Networks and Deep Learning
Intro to Neural Networks and Deep Learning Jack Lanchantin Dr. Yanjun Qi UVA CS 6316 1 Neurons 1-Layer Neural Network Multi-layer Neural Network Loss Functions Backpropagation Nonlinearity Functions NNs
More informationRevision: Neural Network
Revision: Neural Network Exercise 1 Tell whether each of the following statements is true or false by checking the appropriate box. Statement True False a) A perceptron is guaranteed to perfectly learn
More informationCourse 395: Machine Learning - Lectures
Course 395: Machine Learning - Lectures Lecture 1-2: Concept Learning (M. Pantic) Lecture 3-4: Decision Trees & CBC Intro (M. Pantic & S. Petridis) Lecture 5-6: Evaluating Hypotheses (S. Petridis) Lecture
More informationNeural networks. Chapter 19, Sections 1 5 1
Neural networks Chapter 19, Sections 1 5 Chapter 19, Sections 1 5 1 Outline Brains Neural networks Perceptrons Multilayer perceptrons Applications of neural networks Chapter 19, Sections 1 5 2 Brains 10
More informationNeural Networks. Intro to AI Bert Huang Virginia Tech
Neural Networks Intro to AI Bert Huang Virginia Tech Outline Biological inspiration for artificial neural networks Linear vs. nonlinear functions Learning with neural networks: back propagation https://en.wikipedia.org/wiki/neuron#/media/file:chemical_synapse_schema_cropped.jpg
More informationMachine Learning. Neural Networks. Le Song. CSE6740/CS7641/ISYE6740, Fall Lecture 7, September 11, 2012 Based on slides from Eric Xing, CMU
Machine Learning CSE6740/CS7641/ISYE6740, Fall 2012 Neural Networks Le Song Lecture 7, September 11, 2012 Based on slides from Eric Xing, CMU Reading: Chap. 5 CB Learning highly non-linear functions f:
More informationLecture 4: Perceptrons and Multilayer Perceptrons
Lecture 4: Perceptrons and Multilayer Perceptrons Cognitive Systems II - Machine Learning SS 2005 Part I: Basic Approaches of Concept Learning Perceptrons, Artificial Neuronal Networks Lecture 4: Perceptrons
More informationLecture 17: Neural Networks and Deep Learning
UVA CS 6316 / CS 4501-004 Machine Learning Fall 2016 Lecture 17: Neural Networks and Deep Learning Jack Lanchantin Dr. Yanjun Qi 1 Neurons 1-Layer Neural Network Multi-layer Neural Network Loss Functions
More informationNeural Networks with Applications to Vision and Language. Feedforward Networks. Marco Kuhlmann
Neural Networks with Applications to Vision and Language Feedforward Networks Marco Kuhlmann Feedforward networks Linear separability x 2 x 2 0 1 0 1 0 0 x 1 1 0 x 1 linearly separable not linearly separable
More informationAN INTRODUCTION TO NEURAL NETWORKS. Scott Kuindersma November 12, 2009
AN INTRODUCTION TO NEURAL NETWORKS Scott Kuindersma November 12, 2009 SUPERVISED LEARNING We are given some training data: We must learn a function If y is discrete, we call it classification If it is
More informationCSE 417T: Introduction to Machine Learning. Final Review. Henry Chai 12/4/18
CSE 417T: Introduction to Machine Learning Final Review Henry Chai 12/4/18 Overfitting Overfitting is fitting the training data more than is warranted Fitting noise rather than signal 2 Estimating! "#$
More informationThe error-backpropagation algorithm is one of the most important and widely used (and some would say wildly used) learning techniques for neural
1 2 The error-backpropagation algorithm is one of the most important and widely used (and some would say wildly used) learning techniques for neural networks. First we will look at the algorithm itself
More informationMachine Learning (CSE 446): Backpropagation
Machine Learning (CSE 446): Backpropagation Noah Smith c 2017 University of Washington nasmith@cs.washington.edu November 8, 2017 1 / 32 Neuron-Inspired Classifiers correct output y n L n loss hidden units
More informationTHE MOST IMPORTANT BIT
NEURAL NETWORKS THE MOST IMPORTANT BIT A neural network represents a function f : R d R d 2. Peter Orbanz Applied Data Mining 262 BUILDING BLOCKS Units The basic building block is a node or unit: φ The
More informationStatistical Machine Learning from Data
January 17, 2006 Samy Bengio Statistical Machine Learning from Data 1 Statistical Machine Learning from Data Multi-Layer Perceptrons Samy Bengio IDIAP Research Institute, Martigny, Switzerland, and Ecole
More informationLab 5: 16 th April Exercises on Neural Networks
Lab 5: 16 th April 01 Exercises on Neural Networks 1. What are the values of weights w 0, w 1, and w for the perceptron whose decision surface is illustrated in the figure? Assume the surface crosses the
More informationMidterm: CS 6375 Spring 2018
Midterm: CS 6375 Spring 2018 The exam is closed book (1 cheat sheet allowed). Answer the questions in the spaces provided on the question sheets. If you run out of room for an answer, use an additional
More informationCSC321 Lecture 6: Backpropagation
CSC321 Lecture 6: Backpropagation Roger Grosse Roger Grosse CSC321 Lecture 6: Backpropagation 1 / 21 Overview We ve seen that multilayer neural networks are powerful. But how can we actually learn them?
More informationARTIFICIAL INTELLIGENCE. Artificial Neural Networks
INFOB2KI 2017-2018 Utrecht University The Netherlands ARTIFICIAL INTELLIGENCE Artificial Neural Networks Lecturer: Silja Renooij These slides are part of the INFOB2KI Course Notes available from www.cs.uu.nl/docs/vakken/b2ki/schema.html
More information1 What a Neural Network Computes
Neural Networks 1 What a Neural Network Computes To begin with, we will discuss fully connected feed-forward neural networks, also known as multilayer perceptrons. A feedforward neural network consists
More informationBackpropagation: The Good, the Bad and the Ugly
Backpropagation: The Good, the Bad and the Ugly The Norwegian University of Science and Technology (NTNU Trondheim, Norway keithd@idi.ntnu.no October 3, 2017 Supervised Learning Constant feedback from
More informationA summary of Deep Learning without Poor Local Minima
A summary of Deep Learning without Poor Local Minima by Kenji Kawaguchi MIT oral presentation at NIPS 2016 Learning Supervised (or Predictive) learning Learn a mapping from inputs x to outputs y, given
More informationLecture 7 Artificial neural networks: Supervised learning
Lecture 7 Artificial neural networks: Supervised learning Introduction, or how the brain works The neuron as a simple computing element The perceptron Multilayer neural networks Accelerated learning in
More informationNonlinear Classification
Nonlinear Classification INFO-4604, Applied Machine Learning University of Colorado Boulder October 5-10, 2017 Prof. Michael Paul Linear Classification Most classifiers we ve seen use linear functions
More informationNeural Networks. William Cohen [pilfered from: Ziv; Geoff Hinton; Yoshua Bengio; Yann LeCun; Hongkak Lee - NIPs 2010 tutorial ]
Neural Networks William Cohen 10-601 [pilfered from: Ziv; Geoff Hinton; Yoshua Bengio; Yann LeCun; Hongkak Lee - NIPs 2010 tutorial ] WHAT ARE NEURAL NETWORKS? William s notation Logis;c regression + 1
More informationArtificial Neural Networks (ANN) Xiaogang Su, Ph.D. Department of Mathematical Science University of Texas at El Paso
Artificial Neural Networks (ANN) Xiaogang Su, Ph.D. Department of Mathematical Science University of Texas at El Paso xsu@utep.edu Fall, 2018 Outline Introduction A Brief History ANN Architecture Terminology
More informationOnline Videos FERPA. Sign waiver or sit on the sides or in the back. Off camera question time before and after lecture. Questions?
Online Videos FERPA Sign waiver or sit on the sides or in the back Off camera question time before and after lecture Questions? Lecture 1, Slide 1 CS224d Deep NLP Lecture 4: Word Window Classification
More informationLogistic Regression & Neural Networks
Logistic Regression & Neural Networks CMSC 723 / LING 723 / INST 725 Marine Carpuat Slides credit: Graham Neubig, Jacob Eisenstein Logistic Regression Perceptron & Probabilities What if we want a probability
More information