Project 1: A comparison of time delay neural networks (TDNN) trained with mean squared error (MSE) and error entropy criterion (EEC)
|
|
- Thomasine Hall
- 5 years ago
- Views:
Transcription
1 1 Project 1: A comparison of time delay neural networks (TDNN) trained with mean squared error (MSE) and error entropy criterion (EEC) Stefan Craciun Abstract The goal is to implement a TDNN (time delay neural network) trained with both MSE (steepest descent or LMS) and EEC (steepest descent/mee-rip or MEE-SIG) to predict the next sample of the sun spot time series data. TDNN is a nonlinear system, so you will be able to quantify its advantage to predict this difficult time series. Use an input layer with 5 delays as before. The TDNN will have the topology 5-X-1 where the output processing element should be linear (with a bias). You have to select the number of nonlinear (tanh) hidden processing elements X: start with 3 and do not go beyond 10, and choose the one that produces the best results in a cross validation set. If you have used downsampling of the sun spot TS before, keep it the same here such that we can compare the prediction results across linear, nonlinear, MSE and EEC. The nonlinear function used for the PEs is the hyperbolic tangent: 1 2 e 2 + α where n gives the steepness of the nonlinearity. The figure below shows how the slope of the hyperbolic tangent changes as a function of α. This deps on how steep the nonlinearity will be. For this TDNN α is chosen to be F irst the TDNN is built using an input tap delay line with four delays. The network has an input layer of five nonlinear PEs, a hidden layer of nonlinear PEs that will be varied in size and a linear output layer of one PE. The hidden layer will vary from 3 PEs to 10 PEs. Bellow is a figure showing the topology of the TDNN. Hyperbolic tangent function The last PE is a linear PE with a bias so that the output of the TDNN can have a larger set of values not just± 1. TDNN topology with one hidden layer The sun spots data is divided into a training set and a test set using 500 data points to train and 200 points to test. The back-propagation algorithm is implemented in Matlab using incremental or on-line training. This means that for each input vector the error is calculated and back-propagated to update the weights. The back-propagation algorithm is very sensitive to the learning rate parameter as well as to the initial vector weights. It is possible to choose a learning rate for which the back-propagation algorithm will not converge to a stable weight value. It is also observed that for very small step sizes the larger network will take very long to train (error decreases very slowly). The gradient descent method is very slow and so a new cost function is implemented in order to A
2 2 speed up the process. The weights are updated using the gradient descent with momentum. ( 1 α )( η ) + α k 1 ωk = J ω where α=0.9 and the step size η = Both the MSE cost function and the MEE-RIP will update the weights using the steepest descent method with momentum. In order to compare the MSE cost function and the MEE cost function the same training and testing data sets are used as well as the same number of iterations. The graph bellow shows the learning rates for two different topologies. The errors decrease for both the MSE and MEE as the network is trained to predict the time series. Immediately it is observed that a nonlinear predictor such as this TDNN is more accurate in predicting the sun spots time series then a nonlinear network such as the Wiener FIR filter. Here the values of the errors after 3000 iteration decrease below 0.1. In the LMS or Wiener linear filter the errors were on the order of 0.3 after 3000 iterations. The figure below shows the MSE decreasing as a function of time (iterations). 1 hidden layer with 8 PE s using MSE cost function (large step size) 1 hidden layer with 8 PE s using MEE-RIP cost function (large step size) 1 hidden layer with 3 PE s using MSE cost function (large step size) 1 hidden layer with 3 PE s using MEE-RIP cost function (large step size) In order to analyze the performance of each TDNN with MEE-RIP cost function and MSE cost function we quantify the errors looking at both the entropy and the MSE values. The table below summarizes the normalized MSE for both cost functions: Number of hidden PE s MSE normalized MSE MEE-RIP normalized MSE As expected the MSE criterion minimizes the overall MSE so it outperforms the MEE-RIP cost function if we compare normalized MSE as our criterion. However, if we look at a histograms of the errors for a network with 3 hidden PE s there is a significant difference. The TDNN when trained with MEE cost function, trains the network better because it has very few large errors and a majority of small errors. On the other hand the MSE cost function will train the TDNN to approximate the sun spots time series with many
3 3 more larger errors. This can be observed in the figure below. Notice how the MEE cost function has errors with larger variance while the PDF of the errors for the MEE has a higher peak around zero. We will now proceed to test the performance of the of the 2 TDNNs one trained with MSE and one trained with MEE and by building the topology shown in the figure below. PDF of errors when TDNN is trained with MEE-RIP autonomous TDNN PDF of errors when TDNN is trained with MSE Also by looking at the Information Potential as a function of time we notice that the IP does not reach the value of 1 but rather oscillates around 0.7. The figure below shows the normalized IP trained with MEE-RIP. The starting five inputs of the TDNN labeled x1 through x5 are the first five samples of the time series. Instead of using the following sun spot time series such as x6, x7, x100 for the tap delay inputs the output of the network is fed back as the next input. This is called autonomous prediction. Both TDNNs are expected to slowly diverge from the time series. When the normalized error becomes greater than 0.8 the test is stopped. The winner TDNN will be the one which can predict the sun spot time series for the longest time. The results of the autonomous prediction are shown below. Normalized IP trained with MEE-RIP TDNN trained with back-propagation using MEE-RIP
4 4 TDNN trained with back-propagation using MSE It is obvious just by looking at the figures above that the MEE is a slightly better cost function. The TDNN trained with MEE can approximate the sun spots time series for a longer time. The TDNN trained with MSE will diverge much faster from the time series. It seems that once a sample with a large error is predicted the next approximation will be even worse and the predictor will diverge from its goal. The MSE criterion will thus diverge faster because the probability of predicting an erroneous value that is far from the expected time series is high. The MEE is more likely to stumble on a high error and diverge rather than the MEE TDNN. Even when comparing the normalized error the TDNN trained with MEE will reach a normalized MSE greater than 0.8 after 15 samples. The TDNN trained with MEE will reach a normalized MSE greater that 0.8 after 23 samples.
5 5 Matlab Code: MEE Cost Function clear all; load sun_spots.mat; %window_size=100; %filter_order = 5 %initialize the weights w=zeros(1,5); %set step size mu= ; % form the input matrix for row=1:100 for col=1:5 in(row,col)=sun_spots(row+col-1); %form the desired matrix d(i)=sun_spots(i+5); for l=1:200 %% %%%%%%%%%%%% %% %%%%%%%%%%%% %form the error matrix e(i)=d(i)-(in(i,:)*w'); %form the error distances for j=1:100 E(i,j)=e(i)-e(j); %form the input difference matrix for every tap xk(n-i)-xk(n-j) for k=1 for j=1:100 X(j,i)=in(j,k)-in(i,k); A=X; for k=2:5 for j=1:100 X(j,i)=in(j,k)-in(i,k); A=[A;X]; %Use Silverman's rule sig=std(e); sig_opt=2*sig*((1/size(e,2))*1.333)^(1/5); %Calculate the Gaussian of each error distance G=0; for i=1:size(e,1) for j=1:size(e,1) G(i,j)=1/ (sqrt(2*pi)*sig_opt)*exp(-e(i,j)^2/ (2*sig_opt^2)); %Calculate the Gradient Information Potential dv(i)/dw for each tap (10) dv=zeros(1,5); for k=1:5 for i=1:size(e,1) for j=1:size(e,1) dv(k) = dv(k) + G(i,j)*E(i,j)*A(i+k* ,j); dv(k)=-2*dv(k)/100^2; for k=1:5 w(k)=w(k) + mu*dv(k); Backpropagation: for k=0:1000
6 6 %calculate output (activation) %calculate output at first hidden layer sum_hidden_out=weights1*input; for i=1:4 for j=1:8 out_hidden(i,j)=sig(sum_hidde n_out(i,j)); %sigmoid function of W*X %calculate final output sum_out=weights2*out_hidden; out(1,i)=sig(sum_out(1,i)); %sigmoid function of W*Y1 %calculate final error due to each possible input error=desired-out; %backpropagate error at the first hidden layer error_hidden=weights2'*error; %update weights2 %w51 d_weights2(1,1)=0; d_weights2(1,1)= d_weights2(1,1) + t_hidden(1,i); weights2(1,1)= weights2(1,1)+ d_weights2(1,1); %w52 d_weights2(1,2)=0; d_weights2(1,2) = d_weights2(1,2) + t_hidden(2,i); weights2(1,2)= weights2(1,2)+ d_weights2(1,2); %w53 d_weights2(1,3)=0; d_weights2(1,3)=d_weights2(1,3) + t_hidden(3,i); weights2(1,3)= weights2(1,3)+ d_weights2(1,3); %w54 d_weights2(1,4)=0; d_weights2(1,4)=d_weights2(1,4) + t_hidden(4,i); weights2(1,4)= weights2(1,4)+ d_weights2(1,4); %w55 d_weights2(1,5)=0; weights2(1,5)=weights2(1,5) + learning_rate*error(1,i)*der(out(1,i)); weights2(1,5)= weights2(1,5)+ d_weights2(1,5); %update weights1 %w11 d_weights1(1,1)=0; d_weights1(1,1) = d_weights1(1,1) + learning_rate*error_hidden(1,i)*der(out_h idden(1,i))*input(1,i); weights1(1,1) = weights1(1,1)+d_weights1(1,1); %w12 d_weights1(1,2)=0; d_weights1(1,2) = d_weights1(1,2) + learning_rate*error_hidden(1,i)*der(out_h idden(1,i))*input(2,i); weights1(1,2) = weights1(1,2)+d_weights1(1,2); %w13 d_weights1(1,3)=0; d_weights1(1,3) = d_weights1(1,3) + learning_rate*error_hidden(1,i)*der(out_h idden(1,i)); weights1(1,3) = weights1(1,3)+d_weights1(1,3); %w21 d_weights1(2,1)=0;
7 7 d_weights1(2,1) = d_weights1(2,1) + learning_rate*error_hidden(2,i)*der(out_h idden(2,i))*input(1,i); weights1(2,1) = weights1(2,1)+d_weights1(2,1); %w22 d_weights1(2,2)=0; d_weights1(2,2) = d_weights1(2,2) + learning_rate*error_hidden(2,i)*der(out_h idden(2,i))*input(2,i); weights1(2,2) = weights1(2,2)+d_weights1(2,2); %w23 d_weights1(2,3)=0; d_weights1(2,3) = d_weights1(2,3) + learning_rate*error_hidden(2,i)*der(out_h idden(2,i)); weights1(2,3) = weights1(2,3)+d_weights1(2,3); %w31 d_weights1(3,1)=0; d_weights1(3,1) = weights1(3,1) + learning_rate*error_hidden(3,i)*der(out_h idden(3,i))*input(1,i); d_weights1(4,1) = d_weights1(4,1) + learning_rate*error_hidden(4,i)*der(out_h idden(4,i))*input(1,i); weights1(4,1) = weights1(4,1)+d_weights1(4,1); %w42 d_weights1(4,2)=0; d_weights1(4,2) = d_weights1(4,2) + learning_rate*error_hidden(4,i)*der(out_h idden(4,i))*input(2,i); weights1(4,2) = weights1(4,2)+d_weights1(4,2); %w43 d_weights1(4,3)=0; d_weights1(4,3) = d_weights1(4,3) + learning_rate*error_hidden(4,i)*der(out_h idden(4,i)); weights1(4,3) = weights1(4,3)+d_weights1(4,3); %w32 d_weights1(3,2)=0; d_weights1(3,2) = d_weights1(3,2) + learning_rate*error_hidden(3,i)*der(out_h idden(3,i))*input(2,i); weights1(3,2) = weights1(3,2)+d_weights1(3,2); %w33 d_weights1(3,3)=0; d_eights1(3,3) = d_weights1(3,3) + learning_rate*error_hidden(3,i)*der(out_h idden(3,i)); weights1(3,3) = weights1(3,3)+d_weights1(3,3); %w41 d_weights1(4,1)=0;
4. Multilayer Perceptrons
4. Multilayer Perceptrons This is a supervised error-correction learning algorithm. 1 4.1 Introduction A multilayer feedforward network consists of an input layer, one or more hidden layers, and an output
More informationepochs epochs
Neural Network Experiments To illustrate practical techniques, I chose to use the glass dataset. This dataset has 214 examples and 6 classes. Here are 4 examples from the original dataset. The last values
More informationy(x n, w) t n 2. (1)
Network training: Training a neural network involves determining the weight parameter vector w that minimizes a cost function. Given a training set comprising a set of input vector {x n }, n = 1,...N,
More informationSerious limitations of (single-layer) perceptrons: Cannot learn non-linearly separable tasks. Cannot approximate (learn) non-linear functions
BACK-PROPAGATION NETWORKS Serious limitations of (single-layer) perceptrons: Cannot learn non-linearly separable tasks Cannot approximate (learn) non-linear functions Difficult (if not impossible) to design
More informationDeep Learning II: Momentum & Adaptive Step Size
Deep Learning II: Momentum & Adaptive Step Size CS 760: Machine Learning Spring 2018 Mark Craven and David Page www.biostat.wisc.edu/~craven/cs760 1 Goals for the Lecture You should understand the following
More informationCSC 578 Neural Networks and Deep Learning
CSC 578 Neural Networks and Deep Learning Fall 2018/19 3. Improving Neural Networks (Some figures adapted from NNDL book) 1 Various Approaches to Improve Neural Networks 1. Cost functions Quadratic Cross
More informationNeural Networks, Computation Graphs. CMSC 470 Marine Carpuat
Neural Networks, Computation Graphs CMSC 470 Marine Carpuat Binary Classification with a Multi-layer Perceptron φ A = 1 φ site = 1 φ located = 1 φ Maizuru = 1 φ, = 2 φ in = 1 φ Kyoto = 1 φ priest = 0 φ
More informationArtificial Neural Network
Artificial Neural Network Eung Je Woo Department of Biomedical Engineering Impedance Imaging Research Center (IIRC) Kyung Hee University Korea ejwoo@khu.ac.kr Neuron and Neuron Model McCulloch and Pitts
More informationNeural Networks (Part 1) Goals for the lecture
Neural Networks (Part ) Mark Craven and David Page Computer Sciences 760 Spring 208 www.biostat.wisc.edu/~craven/cs760/ Some of the slides in these lectures have been adapted/borrowed from materials developed
More informationLecture 4: Perceptrons and Multilayer Perceptrons
Lecture 4: Perceptrons and Multilayer Perceptrons Cognitive Systems II - Machine Learning SS 2005 Part I: Basic Approaches of Concept Learning Perceptrons, Artificial Neuronal Networks Lecture 4: Perceptrons
More informationIntroduction to Machine Learning
Introduction to Machine Learning Neural Networks Varun Chandola x x 5 Input Outline Contents February 2, 207 Extending Perceptrons 2 Multi Layered Perceptrons 2 2. Generalizing to Multiple Labels.................
More informationBack-Propagation Algorithm. Perceptron Gradient Descent Multilayered neural network Back-Propagation More on Back-Propagation Examples
Back-Propagation Algorithm Perceptron Gradient Descent Multilayered neural network Back-Propagation More on Back-Propagation Examples 1 Inner-product net =< w, x >= w x cos(θ) net = n i=1 w i x i A measure
More informationCSE 190 Fall 2015 Midterm DO NOT TURN THIS PAGE UNTIL YOU ARE TOLD TO START!!!!
CSE 190 Fall 2015 Midterm DO NOT TURN THIS PAGE UNTIL YOU ARE TOLD TO START!!!! November 18, 2015 THE EXAM IS CLOSED BOOK. Once the exam has started, SORRY, NO TALKING!!! No, you can t even say see ya
More informationLinear Neural Networks
Chapter 10 Linear Neural Networks In this chapter, we introduce the concept of the linear neural network. 10.1 Introduction and Notation 1. The linear neural cell, or node has the schematic form as shown
More informationNeural Networks. Yan Shao Department of Linguistics and Philology, Uppsala University 7 December 2016
Neural Networks Yan Shao Department of Linguistics and Philology, Uppsala University 7 December 2016 Outline Part 1 Introduction Feedforward Neural Networks Stochastic Gradient Descent Computational Graph
More informationClassification goals: Make 1 guess about the label (Top-1 error) Make 5 guesses about the label (Top-5 error) No Bounding Box
ImageNet Classification with Deep Convolutional Neural Networks Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton Motivation Classification goals: Make 1 guess about the label (Top-1 error) Make 5 guesses
More informationMultilayer Perceptrons and Backpropagation
Multilayer Perceptrons and Backpropagation Informatics 1 CG: Lecture 7 Chris Lucas School of Informatics University of Edinburgh January 31, 2017 (Slides adapted from Mirella Lapata s.) 1 / 33 Reading:
More informationEngineering Part IIB: Module 4F10 Statistical Pattern Processing Lecture 5: Single Layer Perceptrons & Estimating Linear Classifiers
Engineering Part IIB: Module 4F0 Statistical Pattern Processing Lecture 5: Single Layer Perceptrons & Estimating Linear Classifiers Phil Woodland: pcw@eng.cam.ac.uk Michaelmas 202 Engineering Part IIB:
More informationArtificial Neural Networks. Edward Gatt
Artificial Neural Networks Edward Gatt What are Neural Networks? Models of the brain and nervous system Highly parallel Process information much more like the brain than a serial computer Learning Very
More informationLecture 5: Logistic Regression. Neural Networks
Lecture 5: Logistic Regression. Neural Networks Logistic regression Comparison with generative models Feed-forward neural networks Backpropagation Tricks for training neural networks COMP-652, Lecture
More informationECE521 Lectures 9 Fully Connected Neural Networks
ECE521 Lectures 9 Fully Connected Neural Networks Outline Multi-class classification Learning multi-layer neural networks 2 Measuring distance in probability space We learnt that the squared L2 distance
More informationMultilayer Perceptrons (MLPs)
CSE 5526: Introduction to Neural Networks Multilayer Perceptrons (MLPs) 1 Motivation Multilayer networks are more powerful than singlelayer nets Example: XOR problem x 2 1 AND x o x 1 x 2 +1-1 o x x 1-1
More informationAdaptive Filtering Part II
Adaptive Filtering Part II In previous Lecture we saw that: Setting the gradient of cost function equal to zero, we obtain the optimum values of filter coefficients: (Wiener-Hopf equation) Adaptive Filtering,
More informationNeural networks III: The delta learning rule with semilinear activation function
Neural networks III: The delta learning rule with semilinear activation function The standard delta rule essentially implements gradient descent in sum-squared error for linear activation functions. We
More informationStatistical Machine Learning from Data
January 17, 2006 Samy Bengio Statistical Machine Learning from Data 1 Statistical Machine Learning from Data Multi-Layer Perceptrons Samy Bengio IDIAP Research Institute, Martigny, Switzerland, and Ecole
More informationArtifical Neural Networks
Neural Networks Artifical Neural Networks Neural Networks Biological Neural Networks.................................. Artificial Neural Networks................................... 3 ANN Structure...........................................
More informationArtificial Neural Networks
Artificial Neural Networks Oliver Schulte - CMPT 310 Neural Networks Neural networks arise from attempts to model human/animal brains Many models, many claims of biological plausibility We will focus on
More informationNeural Networks Task Sheet 2. Due date: May
Neural Networks 2007 Task Sheet 2 1/6 University of Zurich Prof. Dr. Rolf Pfeifer, pfeifer@ifi.unizh.ch Department of Informatics, AI Lab Matej Hoffmann, hoffmann@ifi.unizh.ch Andreasstrasse 15 Marc Ziegler,
More informationLecture 10. Neural networks and optimization. Machine Learning and Data Mining November Nando de Freitas UBC. Nonlinear Supervised Learning
Lecture 0 Neural networks and optimization Machine Learning and Data Mining November 2009 UBC Gradient Searching for a good solution can be interpreted as looking for a minimum of some error (loss) function
More informationGradient Descent Training Rule: The Details
Gradient Descent Training Rule: The Details 1 For Perceptrons The whole idea behind gradient descent is to gradually, but consistently, decrease the output error by adjusting the weights. The trick is
More informationNeuro-Fuzzy Comp. Ch. 4 March 24, R p
4 Feedforward Multilayer Neural Networks part I Feedforward multilayer neural networks (introduced in sec 17) with supervised error correcting learning are used to approximate (synthesise) a non-linear
More informationMark Gales October y (x) x 1. x 2 y (x) Inputs. Outputs. x d. y (x) Second Output layer layer. layer.
University of Cambridge Engineering Part IIB & EIST Part II Paper I0: Advanced Pattern Processing Handouts 4 & 5: Multi-Layer Perceptron: Introduction and Training x y (x) Inputs x 2 y (x) 2 Outputs x
More informationSpeaker Representation and Verification Part II. by Vasileios Vasilakakis
Speaker Representation and Verification Part II by Vasileios Vasilakakis Outline -Approaches of Neural Networks in Speaker/Speech Recognition -Feed-Forward Neural Networks -Training with Back-propagation
More informationArtificial Neural Networks
Artificial Neural Networks Stephan Dreiseitl University of Applied Sciences Upper Austria at Hagenberg Harvard-MIT Division of Health Sciences and Technology HST.951J: Medical Decision Support Knowledge
More informationSolutions. Part I Logistic regression backpropagation with a single training example
Solutions Part I Logistic regression backpropagation with a single training example In this part, you are using the Stochastic Gradient Optimizer to train your Logistic Regression. Consequently, the gradients
More informationIntroduction to Natural Computation. Lecture 9. Multilayer Perceptrons and Backpropagation. Peter Lewis
Introduction to Natural Computation Lecture 9 Multilayer Perceptrons and Backpropagation Peter Lewis 1 / 25 Overview of the Lecture Why multilayer perceptrons? Some applications of multilayer perceptrons.
More informationReading Group on Deep Learning Session 1
Reading Group on Deep Learning Session 1 Stephane Lathuiliere & Pablo Mesejo 2 June 2016 1/31 Contents Introduction to Artificial Neural Networks to understand, and to be able to efficiently use, the popular
More informationMachine Learning (CSE 446): Neural Networks
Machine Learning (CSE 446): Neural Networks Noah Smith c 2017 University of Washington nasmith@cs.washington.edu November 6, 2017 1 / 22 Admin No Wednesday office hours for Noah; no lecture Friday. 2 /
More informationChapter 4 Neural Networks in System Identification
Chapter 4 Neural Networks in System Identification Gábor HORVÁTH Department of Measurement and Information Systems Budapest University of Technology and Economics Magyar tudósok körútja 2, 52 Budapest,
More informationEngineering Part IIB: Module 4F10 Statistical Pattern Processing Lecture 6: Multi-Layer Perceptrons I
Engineering Part IIB: Module 4F10 Statistical Pattern Processing Lecture 6: Multi-Layer Perceptrons I Phil Woodland: pcw@eng.cam.ac.uk Michaelmas 2012 Engineering Part IIB: Module 4F10 Introduction In
More informationDay 3 Lecture 3. Optimizing deep networks
Day 3 Lecture 3 Optimizing deep networks Convex optimization A function is convex if for all α [0,1]: f(x) Tangent line Examples Quadratics 2-norms Properties Local minimum is global minimum x Gradient
More informationLab 5: 16 th April Exercises on Neural Networks
Lab 5: 16 th April 01 Exercises on Neural Networks 1. What are the values of weights w 0, w 1, and w for the perceptron whose decision surface is illustrated in the figure? Assume the surface crosses the
More informationNeural Networks with Applications to Vision and Language. Feedforward Networks. Marco Kuhlmann
Neural Networks with Applications to Vision and Language Feedforward Networks Marco Kuhlmann Feedforward networks Linear separability x 2 x 2 0 1 0 1 0 0 x 1 1 0 x 1 linearly separable not linearly separable
More informationRecursive Least Squares for an Entropy Regularized MSE Cost Function
Recursive Least Squares for an Entropy Regularized MSE Cost Function Deniz Erdogmus, Yadunandana N. Rao, Jose C. Principe Oscar Fontenla-Romero, Amparo Alonso-Betanzos Electrical Eng. Dept., University
More informationNeural Network Training
Neural Network Training Sargur Srihari Topics in Network Training 0. Neural network parameters Probabilistic problem formulation Specifying the activation and error functions for Regression Binary classification
More informationMachine Learning and Data Mining. Multi-layer Perceptrons & Neural Networks: Basics. Prof. Alexander Ihler
+ Machine Learning and Data Mining Multi-layer Perceptrons & Neural Networks: Basics Prof. Alexander Ihler Linear Classifiers (Perceptrons) Linear Classifiers a linear classifier is a mapping which partitions
More informationMulti-layer Neural Networks
Multi-layer Neural Networks Steve Renals Informatics 2B Learning and Data Lecture 13 8 March 2011 Informatics 2B: Learning and Data Lecture 13 Multi-layer Neural Networks 1 Overview Multi-layer neural
More informationFeed-forward Networks Network Training Error Backpropagation Applications. Neural Networks. Oliver Schulte - CMPT 726. Bishop PRML Ch.
Neural Networks Oliver Schulte - CMPT 726 Bishop PRML Ch. 5 Neural Networks Neural networks arise from attempts to model human/animal brains Many models, many claims of biological plausibility We will
More informationSimple neuron model Components of simple neuron
Outline 1. Simple neuron model 2. Components of artificial neural networks 3. Common activation functions 4. MATLAB representation of neural network. Single neuron model Simple neuron model Components
More informationModule 12. Machine Learning. Version 2 CSE IIT, Kharagpur
Module 12 Machine Learning Lesson 39 Neural Networks - III 12.4.4 Multi-Layer Perceptrons In contrast to perceptrons, multilayer networks can learn not only multiple decision boundaries, but the boundaries
More information<Special Topics in VLSI> Learning for Deep Neural Networks (Back-propagation)
Learning for Deep Neural Networks (Back-propagation) Outline Summary of Previous Standford Lecture Universal Approximation Theorem Inference vs Training Gradient Descent Back-Propagation
More informationArtificial Neuron (Perceptron)
9/6/208 Gradient Descent (GD) Hantao Zhang Deep Learning with Python Reading: https://en.wikipedia.org/wiki/gradient_descent Artificial Neuron (Perceptron) = w T = w 0 0 + + w 2 2 + + w d d where
More informationMultilayer Perceptron
Outline Hong Chang Institute of Computing Technology, Chinese Academy of Sciences Machine Learning Methods (Fall 2012) Outline Outline I 1 Introduction 2 Single Perceptron 3 Boolean Function Learning 4
More informationC1.2 Multilayer perceptrons
Supervised Models C1.2 Multilayer perceptrons Luis B Almeida Abstract This section introduces multilayer perceptrons, which are the most commonly used type of neural network. The popular backpropagation
More informationECS171: Machine Learning
ECS171: Machine Learning Lecture 4: Optimization (LFD 3.3, SGD) Cho-Jui Hsieh UC Davis Jan 22, 2018 Gradient descent Optimization Goal: find the minimizer of a function min f (w) w For now we assume f
More information10-701/ Machine Learning, Fall
0-70/5-78 Machine Learning, Fall 2003 Homework 2 Solution If you have questions, please contact Jiayong Zhang .. (Error Function) The sum-of-squares error is the most common training
More informationCOMPUTATIONAL INTELLIGENCE (INTRODUCTION TO MACHINE LEARNING) SS16
COMPUTATIONAL INTELLIGENCE (INTRODUCTION TO MACHINE LEARNING) SS6 Lecture 3: Classification with Logistic Regression Advanced optimization techniques Underfitting & Overfitting Model selection (Training-
More informationCS 6501: Deep Learning for Computer Graphics. Basics of Neural Networks. Connelly Barnes
CS 6501: Deep Learning for Computer Graphics Basics of Neural Networks Connelly Barnes Overview Simple neural networks Perceptron Feedforward neural networks Multilayer perceptron and properties Autoencoders
More informationNeural Networks: Backpropagation
Neural Networks: Backpropagation Seung-Hoon Na 1 1 Department of Computer Science Chonbuk National University 2018.10.25 eung-hoon Na (Chonbuk National University) Neural Networks: Backpropagation 2018.10.25
More informationBackpropagation Neural Net
Backpropagation Neural Net As is the case with most neural networks, the aim of Backpropagation is to train the net to achieve a balance between the ability to respond correctly to the input patterns that
More informationMultilayer Neural Networks. (sometimes called Multilayer Perceptrons or MLPs)
Multilayer Neural Networks (sometimes called Multilayer Perceptrons or MLPs) Linear separability Hyperplane In 2D: w x + w 2 x 2 + w 0 = 0 Feature x 2 = w w 2 x w 0 w 2 Feature 2 A perceptron can separate
More informationAddress for Correspondence
Research Article APPLICATION OF ARTIFICIAL NEURAL NETWORK FOR INTERFERENCE STUDIES OF LOW-RISE BUILDINGS 1 Narayan K*, 2 Gairola A Address for Correspondence 1 Associate Professor, Department of Civil
More informationRegularization in Neural Networks
Regularization in Neural Networks Sargur Srihari 1 Topics in Neural Network Regularization What is regularization? Methods 1. Determining optimal number of hidden units 2. Use of regularizer in error function
More informationNeural Networks. CSE 6363 Machine Learning Vassilis Athitsos Computer Science and Engineering Department University of Texas at Arlington
Neural Networks CSE 6363 Machine Learning Vassilis Athitsos Computer Science and Engineering Department University of Texas at Arlington 1 Perceptrons x 0 = 1 x 1 x 2 z = h w T x Output: z x D A perceptron
More informationMachine Learning (CSE 446): Backpropagation
Machine Learning (CSE 446): Backpropagation Noah Smith c 2017 University of Washington nasmith@cs.washington.edu November 8, 2017 1 / 32 Neuron-Inspired Classifiers correct output y n L n loss hidden units
More informationSTOCHASTIC INFORMATION GRADIENT ALGORITHM BASED ON MAXIMUM ENTROPY DENSITY ESTIMATION. Badong Chen, Yu Zhu, Jinchun Hu and Ming Zhang
ICIC Express Letters ICIC International c 2009 ISSN 1881-803X Volume 3, Number 3, September 2009 pp. 1 6 STOCHASTIC INFORMATION GRADIENT ALGORITHM BASED ON MAXIMUM ENTROPY DENSITY ESTIMATION Badong Chen,
More informationGreedy Layer-Wise Training of Deep Networks
Greedy Layer-Wise Training of Deep Networks Yoshua Bengio, Pascal Lamblin, Dan Popovici, Hugo Larochelle NIPS 2007 Presented by Ahmed Hefny Story so far Deep neural nets are more expressive: Can learn
More informationCOGS Q250 Fall Homework 7: Learning in Neural Networks Due: 9:00am, Friday 2nd November.
COGS Q250 Fall 2012 Homework 7: Learning in Neural Networks Due: 9:00am, Friday 2nd November. For the first two questions of the homework you will need to understand the learning algorithm using the delta
More informationLinear Models for Regression
Linear Models for Regression Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjin@postech.ac.kr
More informationLecture 6. Regression
Lecture 6. Regression Prof. Alan Yuille Summer 2014 Outline 1. Introduction to Regression 2. Binary Regression 3. Linear Regression; Polynomial Regression 4. Non-linear Regression; Multilayer Perceptron
More informationMachine Learning Basics III
Machine Learning Basics III Benjamin Roth CIS LMU München Benjamin Roth (CIS LMU München) Machine Learning Basics III 1 / 62 Outline 1 Classification Logistic Regression 2 Gradient Based Optimization Gradient
More informationIntroduction to Neural Networks
CUONG TUAN NGUYEN SEIJI HOTTA MASAKI NAKAGAWA Tokyo University of Agriculture and Technology Copyright by Nguyen, Hotta and Nakagawa 1 Pattern classification Which category of an input? Example: Character
More informationVariable Learning Rate LMS Based Linear Adaptive Inverse Control *
ISSN 746-7659, England, UK Journal of Information and Computing Science Vol., No. 3, 6, pp. 39-48 Variable Learning Rate LMS Based Linear Adaptive Inverse Control * Shuying ie, Chengjin Zhang School of
More informationMachine Learning CS 4900/5900. Lecture 03. Razvan C. Bunescu School of Electrical Engineering and Computer Science
Machine Learning CS 4900/5900 Razvan C. Bunescu School of Electrical Engineering and Computer Science bunescu@ohio.edu Machine Learning is Optimization Parametric ML involves minimizing an objective function
More informationClassification with Perceptrons. Reading:
Classification with Perceptrons Reading: Chapters 1-3 of Michael Nielsen's online book on neural networks covers the basics of perceptrons and multilayer neural networks We will cover material in Chapters
More informationTemporal Backpropagation for FIR Neural Networks
Temporal Backpropagation for FIR Neural Networks Eric A. Wan Stanford University Department of Electrical Engineering, Stanford, CA 94305-4055 Abstract The traditional feedforward neural network is a static
More informationAN INTRODUCTION TO NEURAL NETWORKS. Scott Kuindersma November 12, 2009
AN INTRODUCTION TO NEURAL NETWORKS Scott Kuindersma November 12, 2009 SUPERVISED LEARNING We are given some training data: We must learn a function If y is discrete, we call it classification If it is
More informationChapter ML:VI (continued)
Chapter ML:VI (continued) VI Neural Networks Perceptron Learning Gradient Descent Multilayer Perceptron Radial asis Functions ML:VI-64 Neural Networks STEIN 2005-2018 Definition 1 (Linear Separability)
More informationLecture 35: Optimization and Neural Nets
Lecture 35: Optimization and Neural Nets CS 4670/5670 Sean Bell DeepDream [Google, Inceptionism: Going Deeper into Neural Networks, blog 2015] Aside: CNN vs ConvNet Note: There are many papers that use
More informationDeep Neural Networks
Deep Neural Networks DT2118 Speech and Speaker Recognition Giampiero Salvi KTH/CSC/TMH giampi@kth.se VT 2015 1 / 45 Outline State-to-Output Probability Model Artificial Neural Networks Perceptron Multi
More informationLinear Regression. CSL603 - Fall 2017 Narayanan C Krishnan
Linear Regression CSL603 - Fall 2017 Narayanan C Krishnan ckn@iitrpr.ac.in Outline Univariate regression Multivariate regression Probabilistic view of regression Loss functions Bias-Variance analysis Regularization
More informationLinear Regression. CSL465/603 - Fall 2016 Narayanan C Krishnan
Linear Regression CSL465/603 - Fall 2016 Narayanan C Krishnan ckn@iitrpr.ac.in Outline Univariate regression Multivariate regression Probabilistic view of regression Loss functions Bias-Variance analysis
More informationAI Programming CS F-20 Neural Networks
AI Programming CS662-2008F-20 Neural Networks David Galles Department of Computer Science University of San Francisco 20-0: Symbolic AI Most of this class has been focused on Symbolic AI Focus or symbols
More informationLinear Least-Squares Based Methods for Neural Networks Learning
Linear Least-Squares Based Methods for Neural Networks Learning Oscar Fontenla-Romero 1, Deniz Erdogmus 2, JC Principe 2, Amparo Alonso-Betanzos 1, and Enrique Castillo 3 1 Laboratory for Research and
More informationCS 343: Artificial Intelligence
CS 343: Artificial Intelligence Deep Learning Prof. Scott Niekum The University of Texas at Austin [These slides based on those of Dan Klein, Pieter Abbeel, Anca Dragan for CS188 Intro to AI at UC Berkeley.
More informationNeural networks. Chapter 20, Section 5 1
Neural networks Chapter 20, Section 5 Chapter 20, Section 5 Outline Brains Neural networks Perceptrons Multilayer perceptrons Applications of neural networks Chapter 20, Section 5 2 Brains 0 neurons of
More informationNeural Network Control
Neural Network Control Daniel Eggert 24th February 2003 Technical University of Denmark Informatics and Mathematical Modelling Building 321, DK-2800 Lyngby, Denmark Phone +45 4525 3351, Fax +45 4588 2673
More informationCSCI 1951-G Optimization Methods in Finance Part 12: Variants of Gradient Descent
CSCI 1951-G Optimization Methods in Finance Part 12: Variants of Gradient Descent April 27, 2018 1 / 32 Outline 1) Moment and Nesterov s accelerated gradient descent 2) AdaGrad and RMSProp 4) Adam 5) Stochastic
More informationNeural networks. Chapter 19, Sections 1 5 1
Neural networks Chapter 19, Sections 1 5 Chapter 19, Sections 1 5 1 Outline Brains Neural networks Perceptrons Multilayer perceptrons Applications of neural networks Chapter 19, Sections 1 5 2 Brains 10
More informationOverview of gradient descent optimization algorithms. HYUNG IL KOO Based on
Overview of gradient descent optimization algorithms HYUNG IL KOO Based on http://sebastianruder.com/optimizing-gradient-descent/ Problem Statement Machine Learning Optimization Problem Training samples:
More informationEPL442: Computational
EPL442: Computational Learning Systems Lab 2 Vassilis Vassiliades Department of Computer Science University of Cyprus Outline Artificial Neuron Feedforward Neural Network Back-propagation Algorithm Notes
More informationNeed for Deep Networks Perceptron. Can only model linear functions. Kernel Machines. Non-linearity provided by kernels
Need for Deep Networks Perceptron Can only model linear functions Kernel Machines Non-linearity provided by kernels Need to design appropriate kernels (possibly selecting from a set, i.e. kernel learning)
More informationIntroduction Neural Networks - Architecture Network Training Small Example - ZIP Codes Summary. Neural Networks - I. Henrik I Christensen
Neural Networks - I Henrik I Christensen Robotics & Intelligent Machines @ GT Georgia Institute of Technology, Atlanta, GA 30332-0280 hic@cc.gatech.edu Henrik I Christensen (RIM@GT) Neural Networks 1 /
More informationCS:4420 Artificial Intelligence
CS:4420 Artificial Intelligence Spring 2018 Neural Networks Cesare Tinelli The University of Iowa Copyright 2004 18, Cesare Tinelli and Stuart Russell a a These notes were originally developed by Stuart
More information1 What a Neural Network Computes
Neural Networks 1 What a Neural Network Computes To begin with, we will discuss fully connected feed-forward neural networks, also known as multilayer perceptrons. A feedforward neural network consists
More informationNeural Networks. Volker Tresp Summer 2015
Neural Networks Volker Tresp Summer 2015 1 Introduction The performance of a classifier or a regression model critically depends on the choice of appropriate basis functions The problem with generic basis
More informationConjugate gradient algorithm for training neural networks
. Introduction Recall that in the steepest-descent neural network training algorithm, consecutive line-search directions are orthogonal, such that, where, gwt [ ( + ) ] denotes E[ w( t + ) ], the gradient
More informationNeural Networks. Bishop PRML Ch. 5. Alireza Ghane. Feed-forward Networks Network Training Error Backpropagation Applications
Neural Networks Bishop PRML Ch. 5 Alireza Ghane Neural Networks Alireza Ghane / Greg Mori 1 Neural Networks Neural networks arise from attempts to model human/animal brains Many models, many claims of
More informationNeural networks. Chapter 20. Chapter 20 1
Neural networks Chapter 20 Chapter 20 1 Outline Brains Neural networks Perceptrons Multilayer networks Applications of neural networks Chapter 20 2 Brains 10 11 neurons of > 20 types, 10 14 synapses, 1ms
More informationNeed for Deep Networks Perceptron. Can only model linear functions. Kernel Machines. Non-linearity provided by kernels
Need for Deep Networks Perceptron Can only model linear functions Kernel Machines Non-linearity provided by kernels Need to design appropriate kernels (possibly selecting from a set, i.e. kernel learning)
More information