ARTIFICIAL NEURAL NETWORK PART I HANIEH BORHANAZAD


 Elmer George
 1 years ago
 Views:
Transcription
1 ARTIFICIAL NEURAL NETWORK PART I HANIEH BORHANAZAD
2 WHAT IS A NEURAL NETWORK? The simplest definition of a neural network, more properly referred to as an 'artificial' neural network (ANN), is provided by the inventor of one of the first neurocomputers, Dr. Robert HechtNielsen. He defines a neural network as: "...a computing system made up of a number of simple, highly interconnected processing elements, which process information by their dynamic state response to external inputs.
3 WHAT ARE ARTIFICIAL NEURAL NETWORKS? The brain basically learns from experience. Now, advances in biological research promise an initial understanding of the natural thinking mechanism. This research shows that brains store information as patterns. Some of these patterns are very complicated and allow us the ability to recognize individual faces from many different angles. This process of storing information as patterns, utilizing those patterns, and then solving problems encompasses a new field in computing. This field, as mentioned before, does not utilize traditional programming but involves the creation of massively parallel networks and the training of those networks to solve specific problems. This field also utilizes words very different from traditional computing, words like behave, react, selforganize, learn, generalize, and forge
4 HOW NEURONS WORK? The design of Artificial Neural Network (ANN) is inspired by human brain. It is important to have a rough idea what is going on in our brain. Therefore, before we look into the artificial one, let s have a look at the real one.
5 WHAT ARE ARTIFICIAL NEURAL NETWORKS? Neuron in ANNs tend to have fewer connections than biological neurons. Each neuron in ANN receives a number of inputs. An activation function is applied to these inputs which results in activation level of neuron (output value of the neuron). Knowledge about the learning task is given in the form of examples called training examples.
6 ARTIFICIAL NEURAL NETWORKS An Artificial Neural Network is specified by: Neuron model: the information processing unit of the NN, An architecture: a set of neurons and links connecting neurons. Each link has a weight, A learning algorithm: used for training the NN by modifying the weights in order to model a particular learning task correctly on the training examples. The aim is to obtain a NN that is trained and generalizes well. It should behaves correctly on new instances of the learning task.
7 THE NEURON DIAGRAM
8 NEURON The neuron is the basic information processing unit of a NN. It consists of: 1 A set of links, describing the neuron inputs, with weights W 1, W 2,, W m 2 An adder function (linear combiner) for computing the weighted sum of the inputs: (real numbers) u m wjxj j 1 3 Activation function for limiting the amplitude of the neuron output. Here b denotes bias. y (u b)
9 BIAS OF A NEURON The bias b has the effect of applying a transformation to the weighted sum u v = u + b The bias is an external parameter of the neuron. It can be modeled by adding an extra input. v is called induced field of the neuron v w m wjx 0 j 0 b j
10 HOW DOES THE NEURON DETERMINE ITS OUTPUT? The neuron computes the weighted sum of the input signals and compares the result with a threshold value, θ. If the net input is less than the threshold, the neuron output is 1. But if the net input is greater than or equal to the threshold, the neuron becomes activated and its output attains a value +1(McCulloch and Pitts, 1943).
11 HOW DOES THE NEURON DETERMINE ITS OUTPUT? In other words, the neuron uses the following transfer or activation function: where X is the net weighted input to the neuron, x i is the value of input i, w i is the weight of input i, n is the number of neuron inputs, and Y is the outputof the neuron. This type of activation function is called a sign function. Thus the actual output of the neuron with a sign activation function can be represented as:
12 IS THE SIGN FUNCTION THE ONLY ACTIVATION FUNCTION USED BY NEURONS? The choice of activation function Examples: 1. step function 2. ramp function 3. sigmoid function 4. Gaussian function determines the neuron model. Note: The step and sign activation functions, also called hard limit functions, are often used in decisionmaking neurons for classification and pattern recognition tasks.
13 Step Function ( v) a b if if v c v c b a c
14 Ramp Function ( v) a b a if if v c v d (( v c)( b a) /( d c)) otherwise b a c d
15 Sigmoid function ( v) z 1 1 exp( xv y)
16 The Gaussian function is the probability function of the normal distribution. Sometimes also called the frequency curve. ( v) 1 exp v 2
17 u m y (u b) wjxj j 1 1. step function 2. ramp function 3. sigmoid function 4. Gaussian function
18 NETWORK ARCHITECTURES Three different classes of network architectures: singlelayer feedforward multilayer feedforward recurrent The architecture of a neural network is linked with the learning algorithm used to train
19 WHAT IS FEEDFORWARD NN? In a feed forward network information always moves one direction; it never goes backwards. A feedforward neural network is an artificial neural network where connections between the units do not form a directed cycle. This is different from recurrent neural networks. The feedforward neural network was the first and simplest type of artificial neural network devised. In this network, the information moves in only one direction, forward, from the input nodes, through the hidden nodes (if any) and to the output nodes. There are no cycles or loops in the network.
20 SINGLELAYER FEEDFORWARD The simplest kind of neural network is a singlelayer perceptron network, which consists of a single layer of output nodes; the inputs are fed directly to the outputs via a series of weights. It can be considered the simplest kind of feedforward network. A perceptron can be created using any values for the activated and deactivated states as long as the threshold value lies between the two.
21 SINGLELAYER FEEDFORWARD The sum of the products of the weights and the inputs is calculated in each node, and if the value is above some threshold (typically 0) the neuron fires and takes the activated value (typically 1); otherwise it takes the deactivated value (typically 1). The output node has a "threshold",v. Rule: If summed input v, then it "fires" (output y = 1). Else (summed input < v) it doesn't fire (output y = 0). b (bias) x 1 x 2 w 1 w 2 v (v) y x n w n ( v) 1if 1if v v 0 0
22 SINGLELAYER LIMITATION The perceptron is used for binary classification. First train a perceptron for a classification task. Find suitable weights in such a way that the training examples are correctly classified. Geometrically try to find a hyperplane that separates the examples of the two classes. The perceptron can only model linearly separable classes. When the two classes are not linearly separable, it may be desirable to obtain a linear separator that minimizes the mean squared error. Given training examples of classes C 1, C 2 train the perceptron in such a way that : If the output of the perceptron is +1 then the input is assigned to class C 1 If the output is 1 then the input is assigned to C 2 Although a single threshold unit is quite limited in its computational power, it has been shown that networks of parallel threshold units can approximate any continuous function from a compact interval of the real numbers into the interval [1,1].
23 MULTI LAYER FEEDFORWARD FFNN is a more general network architecture, where there are hidden layers between input and output layers. Hidden nodes do not directly receive inputs nor send outputs to the external environment. FFNNs overcome the limitation of singlelayer NN. They can handle nonlinearly separable learning tasks. Input layer Output layer Hidden Layer Network
24 CAN A NEURAL NETWORK INCLUDE MORE THAN TWO HIDDEN LAYERS? Commercial ANNs incorporate three and sometimes four layers, including one or two hidden layers. Each layer can contain from 10 to 1000 neurons. Experimental neural networks may have five or even six layers, including three or four hidden layers, and utilise millions of neurons, but most practical applications use only three layers, because each additional layer increases the computational burden exponentially.
25 TRAINING IN SINGLE LAYER The model consists of a linear combiner followed by a hard limiter. The weighted sum of the inputs is applied to the hard limiter, which produces an output equal to  1 if its input is positive and +1 if it is negative. The aim of the perceptron is to classify inputs, or in other words externally applied stimuli x 1 ; x 2 ;... ; x n, into one of two classes, say A 1 and A 2
26 TRAINING IN SINGLE LAYER For the case of two inputs, x 1 and x 2, the decision boundary takes the form of a straight line shown in bold. Point 1, which lies above the boundary line, belongs to class A 1 ; and point 2, which lies below the line, belongs to class A 2. The threshold θ can be used to shift the decision boundary.
27 HOW DOES THE PERCEPTRON LEARN ITS CLASSIfiCATION TASKS? The perceptron is trained (i.e., the weights and threshold values are calculated) based on an iterative training phase involving training data. Training data are composed of a list of input values and their associated desired output values. In the training phase, the inputs and related outputs of the training data are repeatedly submitted to the perceptron. The perceptron calculates an output value for each set of input values.
28 TRAINING IN SINGLE LAYER (EXAMPLE) Example: If the output of a particular training case is labelled 1 when it should be labelled 0, the threshold value (theta) is increased by 1, and all weight values associated with inputs of 1 are decreased by 1. The opposite is performed if the output of a training case is labelled 0 when it should be labelled 1. No changes are made to the threshold value or weights if a particular training case is correctly classified.
29 TRAINING IN SINGLE LAYER (EXAMPLE) This set of training rules is summarized as: If OUTPUT is correct, then no changes are made to the threshold or weights If OUTPUT = 1, but should be 0 then {theta = theta + 1} and {weight x = weight x 1, if input x = 1} If OUTPUT = 0, but should be 1 then {theta = theta  1} and {weight x =weight x +1, if input x = 1} An example of a perceptron. The system consists of binary
30 TRAINING IN SINGLE LAYER (EXAMPLE) 1 An example of a perceptron. The system consists of binary activations. Weights are identified by w s, and inputs are identified by i s. A variable threshold value (theta) is used at the
31 THE PERCEPTRON LEARNING RULE If at iteration p, the actual output is Y(p)and the desired output is Yd(p) then the error is given by: Iteration p here refers to the pth training example presented to the perceptron. If the error, e(p), is positive, we need to increase perceptron output Y(p), but if it is negative, we need to decrease Y(p). Taking into account that each perceptron input contributes x i (p)* w i(p)to the total input X (p), we find that if input value x i(p)is positive, an increase in its weight w i (p)tends to increase perceptron output Y (p), whereas if x i (p) is negative, an increase in w i(p)tends to decrease Y (p). Thus, the following perceptron learning rule can be established: Where α is the learning rate, a positive constant less than unity
32 TRAINING IN SINGLE LAYER Once the network is trained, it can be used to classify new data sets whose input/output associations are similar to those that characterize the training data set. Thus, through an iterative training stage in which the weights and threshold gradually migrate to useful values (i.e., values that minimize or eliminate error), the perceptron can be said to learn how to solve simple problems.
33 EXAMPLE: TRAIN A PERCEPTRON TO PERFORM BASIC LOGICAL OPERATIONS (AND) Truth tables for AND
34 EXAMPLE: TRAIN A PERCEPTRON TO PERFORM BASIC LOGICAL OPERATIONS (AND) The perceptron output Y is 1 only if the total weighted input X is greater than or equal to the threshold value θ. This means that the entire input space is divided in two along a boundary defined by X =θ. If we substitute values for weights w1 and w2 and threshold θ=0.2, we obtain one of the possible separating lines as Truth tables for AND Thus, the region below the boundary line, where the output is 0, is given by and the region above this line, where the output is 1, is given by The fact that a perceptron can learn only linear separable functions is rather bad news, because there are not many such functions.
35 TRAINING IN MULTILAYER More than a hundred different learning algorithms are available to train MLP, but then most popular method is backpropagation. With backpropagation, the input data is repeatedly presented to the neural network. With each presentation the output of the neural network is compared to the desired output and an error is computed. This error is then fed back (backpropagated) to the neural network and used to adjust the weights such that the error decreases with each iteration and the neural model gets closer and closer to producing the desired output. This process is known as "training". During the training session, the neural network receives a number of different input patterns, discovers significant features in these patterns and learns how to classify input data into appropriate categories.
36 TRAINING IN MULTILAYER
37 TRAINING ALGORITHM: BACKPROPAGATION The Backpropagation single perceptron. algorithm learns in the same way as It searches for weight values that minimize the total error of the network over the set of training examples (training set). Backpropagation consists of the repeated application of the following two passes: Forward pass: In this step, the network is activated on one example and the error of (each neuron of) the output layer is computed. Backward pass: in this step the network error is used for updating the weights. The error is propagated backwards from the output layer through the network layer by layer. This is done by recursively computing the local gradient of each neuron.
38 BACKPROPAGATION Backpropagation training algorithm Network activation Forward Step Error propagation Backward Step Backpropagation adjusts the weights of the NN in order to minimize the network total mean squared error.
39 BACKPROPAGATION With one hidden layer, we can represent any continuous function of the input signals, and with two hidden layers even discontinuous functions can be represented. Please go through PDF file
40 NN DESIGN ISSUES Data representation Network Topology Network Parameters Training Validation
41 Data Representation Data representation depends on the problem. In general ANNs work on continuous (real valued) attributes. Therefore symbolic attributes are encoded into continuous ones. Attributes of different types may have different ranges of values which affect the training process. Normalization may be used, like the following one which scales each attribute to assume values between 0 and 1. x i xi min i max min i for each value x i of i th attribute, min i and max i are the minimum and maximum value of that attribute over the training set. i
42 Network Topology The number of layers and neurons depend on the specific task. In practice this issue is solved by trial and error. Two types of adaptive algorithms can be used: start from a large network and successively remove some neurons and links until network performance degrades. begin with a small network and introduce new neurons until performance is satisfactory.
43 Network parameters How are the weights initialized? How is the learning rate chosen? How many hidden layers and how many neurons? How many examples in the training set?
44 INITIALIZATION OF WEIGHTS In general, initial weights are randomly chosen, with typical values between 1.0 and 1.0 or 0.5 and 0.5. If some inputs are much larger than others, random initialization may bias the network to give much more importance to larger inputs. In such a case, weights can be initialized as follows: 1 1 w ij 2N xi i 1,..., N 1 1 w jk 2N ( wijx i 1,..., N i ) For weights from the input to the first layer For weights from the first to the second layer
45 Choice of learning rate The right value of α depends on the application. Values between 0.1 and 0.9 have been used in many applications. Other heuristics is that adapt α during the training as described in previous slides.
46 NUMBER OF TRAINING Rule of thumb: the number of training examples should be at least five to ten times the number of weights of the network. Other rule: N W (1 a) W = number of weights a=expected accuracy on test set
47 RECURRENT NETWORK FFNN is acyclic where data passes from input to the output nodes and not vice versa. Once the FFNN is trained, its state is fixed and does not alter as new data is presented to it. It does not have memory. Recurrent network can have connections that go backward from output to input nodes and models dynamic systems. In this way, a recurrent network s internal state can be altered as sets of input data are presented. It can be said to have memory. It is useful in solving problems where the solution depends not just on the current inputs but on all previous inputs. Applications predict stock market price, weather forecast
48 RECURRENT NETWORK ARCHITECTURE Recurrent Network with hidden neuron: unit delay operator d is used to model a dynamic system d d input hidden output d
49 LEARNING AND TRAINING During learning phase, a recurrent network feeds its inputs through the network, including feeding data back from outputs to inputs process is repeated until the values of the outputs do not change. This state is called equilibrium or stability Recurrent networks can be trained by using backpropagation algorithm. In this method, at each step, the activation of the output is compared with the desired activation and errors are propagated backward through the network. Once this training process is completed, the network becomes capable of performing a sequence of actions.
50 SUMMARY ANN Neuron model Architecture Learning Algorithm step function ramp function sigmoid function Gaussian function recurrent singlelayer feedforward multilayer feedforward backpropagation
51
Lecture 7 Artificial neural networks: Supervised learning
Lecture 7 Artificial neural networks: Supervised learning Introduction, or how the brain works The neuron as a simple computing element The perceptron Multilayer neural networks Accelerated learning in
More informationLecture 7 Artificial neural networks: Supervised learning
Lecture 7 Artificial neural networks: Supervised learning Introduction, or how the brain works The neuron as a simple computing element The perceptron Multilayer neural networks Accelerated learning in
More informationNeural Networks and the Backpropagation Algorithm
Neural Networks and the Backpropagation Algorithm Francisco S. Melo In these notes, we provide a brief overview of the main concepts concerning neural networks and the backpropagation algorithm. We closely
More informationSimple Neural Nets For Pattern Classification
CHAPTER 2 Simple Neural Nets For Pattern Classification Neural Networks General Discussion One of the simplest tasks that neural nets can be trained to perform is pattern classification. In pattern classification
More informationData Mining Part 5. Prediction
Data Mining Part 5. Prediction 5.5. Spring 2010 Instructor: Dr. Masoud Yaghini Outline How the Brain Works Artificial Neural Networks Simple Computing Elements FeedForward Networks Perceptrons (Singlelayer,
More informationArtificial Neural Networks. Part 2
Artificial Neural Netorks Part Artificial Neuron Model Folloing simplified model of real neurons is also knon as a Threshold Logic Unit x McCullouchPitts neuron (943) x x n n Body of neuron f out Biological
More informationARTIFICIAL INTELLIGENCE. Artificial Neural Networks
INFOB2KI 20172018 Utrecht University The Netherlands ARTIFICIAL INTELLIGENCE Artificial Neural Networks Lecturer: Silja Renooij These slides are part of the INFOB2KI Course Notes available from www.cs.uu.nl/docs/vakken/b2ki/schema.html
More informationIntroduction to Neural Networks
Introduction to Neural Networks What are (Artificial) Neural Networks? Models of the brain and nervous system Highly parallel Process information much more like the brain than a serial computer Learning
More informationCourse 395: Machine Learning  Lectures
Course 395: Machine Learning  Lectures Lecture 12: Concept Learning (M. Pantic) Lecture 34: Decision Trees & CBC Intro (M. Pantic & S. Petridis) Lecture 56: Evaluating Hypotheses (S. Petridis) Lecture
More informationArtificial Neural Network
Artificial Neural Network Contents 2 What is ANN? Biological Neuron Structure of Neuron Types of Neuron Models of Neuron Analogy with human NN Perceptron OCR Multilayer Neural Network Back propagation
More informationFeedforward Neural Nets and Backpropagation
Feedforward Neural Nets and Backpropagation Julie Nutini University of British Columbia MLRG September 28 th, 2016 1 / 23 Supervised Learning Roadmap Supervised Learning: Assume that we are given the features
More informationCS:4420 Artificial Intelligence
CS:4420 Artificial Intelligence Spring 2018 Neural Networks Cesare Tinelli The University of Iowa Copyright 2004 18, Cesare Tinelli and Stuart Russell a a These notes were originally developed by Stuart
More informationAI Programming CS F20 Neural Networks
AI Programming CS6622008F20 Neural Networks David Galles Department of Computer Science University of San Francisco 200: Symbolic AI Most of this class has been focused on Symbolic AI Focus or symbols
More information18.6 Regression and Classification with Linear Models
18.6 Regression and Classification with Linear Models 352 The hypothesis space of linear functions of continuousvalued inputs has been used for hundreds of years A univariate linear function (a straight
More information22c145Fall 01: Neural Networks. Neural Networks. Readings: Chapter 19 of Russell & Norvig. Cesare Tinelli 1
Neural Networks Readings: Chapter 19 of Russell & Norvig. Cesare Tinelli 1 Brains as Computational Devices Brains advantages with respect to digital computers: Massively parallel Faulttolerant Reliable
More informationArtificial Neural Networks" and Nonparametric Methods" CMPSCI 383 Nov 17, 2011!
Artificial Neural Networks" and Nonparametric Methods" CMPSCI 383 Nov 17, 2011! 1 Todayʼs lecture" How the brain works (!)! Artificial neural networks! Perceptrons! Multilayer feedforward networks! Error
More informationAN INTRODUCTION TO NEURAL NETWORKS. Scott Kuindersma November 12, 2009
AN INTRODUCTION TO NEURAL NETWORKS Scott Kuindersma November 12, 2009 SUPERVISED LEARNING We are given some training data: We must learn a function If y is discrete, we call it classification If it is
More informationArtificial Intelligence
Artificial Intelligence Jeff Clune Assistant Professor Evolving Artificial Intelligence Laboratory Announcements Be making progress on your projects! Three Types of Learning Unsupervised Supervised Reinforcement
More informationNeural networks. Chapter 20. Chapter 20 1
Neural networks Chapter 20 Chapter 20 1 Outline Brains Neural networks Perceptrons Multilayer networks Applications of neural networks Chapter 20 2 Brains 10 11 neurons of > 20 types, 10 14 synapses, 1ms
More informationNeural Networks. Nicholas Ruozzi University of Texas at Dallas
Neural Networks Nicholas Ruozzi University of Texas at Dallas Handwritten Digit Recognition Given a collection of handwritten digits and their corresponding labels, we d like to be able to correctly classify
More informationMachine Learning. Neural Networks
Machine Learning Neural Networks Bryan Pardo, Northwestern University, Machine Learning EECS 349 Fall 2007 Biological Analogy Bryan Pardo, Northwestern University, Machine Learning EECS 349 Fall 2007 THE
More informationCOGS Q250 Fall Homework 7: Learning in Neural Networks Due: 9:00am, Friday 2nd November.
COGS Q250 Fall 2012 Homework 7: Learning in Neural Networks Due: 9:00am, Friday 2nd November. For the first two questions of the homework you will need to understand the learning algorithm using the delta
More informationIntroduction to Natural Computation. Lecture 9. Multilayer Perceptrons and Backpropagation. Peter Lewis
Introduction to Natural Computation Lecture 9 Multilayer Perceptrons and Backpropagation Peter Lewis 1 / 25 Overview of the Lecture Why multilayer perceptrons? Some applications of multilayer perceptrons.
More informationARTIFICIAL NEURAL NETWORKS گروه مطالعاتي 17 بهار 92
ARTIFICIAL NEURAL NETWORKS گروه مطالعاتي 17 بهار 92 BIOLOGICAL INSPIRATIONS Some numbers The human brain contains about 10 billion nerve cells (neurons) Each neuron is connected to the others through 10000
More informationPart 8: Neural Networks
METU Informatics Institute Min720 Pattern Classification ith BioMedical Applications Part 8: Neural Netors  INTRODUCTION: BIOLOGICAL VS. ARTIFICIAL Biological Neural Netors A Neuron:  A nerve cell as
More informationCSE 352 (AI) LECTURE NOTES Professor Anita Wasilewska. NEURAL NETWORKS Learning
CSE 352 (AI) LECTURE NOTES Professor Anita Wasilewska NEURAL NETWORKS Learning Neural Networks Classifier Short Presentation INPUT: classification data, i.e. it contains an classification (class) attribute.
More informationNeural Networks and Fuzzy Logic Rajendra Dept.of CSE ASCET
Unit. Definition Neural network is a massively parallel distributed processing system, made of highly interconnected neural computing elements that have the ability to learn and thereby acquire knowledge
More informationIntroduction to Artificial Neural Networks
Facultés Universitaires NotreDame de la Paix 27 March 2007 Outline 1 Introduction 2 Fundamentals Biological neuron Artificial neuron Artificial Neural Network Outline 3 Singlelayer ANN Perceptron Adaline
More informationNeural Networks and Ensemble Methods for Classification
Neural Networks and Ensemble Methods for Classification NEURAL NETWORKS 2 Neural Networks A neural network is a set of connected input/output units (neurons) where each connection has a weight associated
More informationNeural networks. Chapter 19, Sections 1 5 1
Neural networks Chapter 19, Sections 1 5 Chapter 19, Sections 1 5 1 Outline Brains Neural networks Perceptrons Multilayer perceptrons Applications of neural networks Chapter 19, Sections 1 5 2 Brains 10
More informationSPSS, University of Texas at Arlington. Topics in Machine LearningEE 5359 Neural Networks
Topics in Machine LearningEE 5359 Neural Networks 1 The Perceptron Output: A perceptron is a function that maps Ddimensional vectors to real numbers. For notational convenience, we add a zeroth dimension
More informationCOMP4360 Machine Learning Neural Networks
COMP4360 Machine Learning Neural Networks Jacky Baltes Autonomous Agents Lab University of Manitoba Winnipeg, Canada R3T 2N2 Email: jacky@cs.umanitoba.ca WWW: http://www.cs.umanitoba.ca/~jacky http://aalab.cs.umanitoba.ca
More informationMultilayer Perceptron
Outline Hong Chang Institute of Computing Technology, Chinese Academy of Sciences Machine Learning Methods (Fall 2012) Outline Outline I 1 Introduction 2 Single Perceptron 3 Boolean Function Learning 4
More informationNeural Networks. Chapter 18, Section 7. TB Artificial Intelligence. Slides from AIMA 1/ 21
Neural Networks Chapter 8, Section 7 TB Artificial Intelligence Slides from AIMA http://aima.cs.berkeley.edu / 2 Outline Brains Neural networks Perceptrons Multilayer perceptrons Applications of neural
More informationArtifical Neural Networks
Neural Networks Artifical Neural Networks Neural Networks Biological Neural Networks.................................. Artificial Neural Networks................................... 3 ANN Structure...........................................
More informationArtificial neural networks
Artificial neural networks Chapter 8, Section 7 Artificial Intelligence, spring 203, Peter Ljunglöf; based on AIMA Slides c Stuart Russel and Peter Norvig, 2004 Chapter 8, Section 7 Outline Brains Neural
More informationCS 6501: Deep Learning for Computer Graphics. Basics of Neural Networks. Connelly Barnes
CS 6501: Deep Learning for Computer Graphics Basics of Neural Networks Connelly Barnes Overview Simple neural networks Perceptron Feedforward neural networks Multilayer perceptron and properties Autoencoders
More informationArtificial Neural Networks. Edward Gatt
Artificial Neural Networks Edward Gatt What are Neural Networks? Models of the brain and nervous system Highly parallel Process information much more like the brain than a serial computer Learning Very
More informationLecture 4: Perceptrons and Multilayer Perceptrons
Lecture 4: Perceptrons and Multilayer Perceptrons Cognitive Systems II  Machine Learning SS 2005 Part I: Basic Approaches of Concept Learning Perceptrons, Artificial Neuronal Networks Lecture 4: Perceptrons
More informationRevision: Neural Network
Revision: Neural Network Exercise 1 Tell whether each of the following statements is true or false by checking the appropriate box. Statement True False a) A perceptron is guaranteed to perfectly learn
More informationArtificial Neural Networks D B M G. Data Base and Data Mining Group of Politecnico di Torino. Elena Baralis. Politecnico di Torino
Artificial Neural Networks Data Base and Data Mining Group of Politecnico di Torino Elena Baralis Politecnico di Torino Artificial Neural Networks Inspired to the structure of the human brain Neurons as
More information(FeedForward) Neural Networks Dr. Hajira Jabeen, Prof. Jens Lehmann
(FeedForward) Neural Networks 20161206 Dr. Hajira Jabeen, Prof. Jens Lehmann Outline In the previous lectures we have learned about tensors and factorization methods. RESCAL is a bilinear model for
More informationAn artificial neural networks (ANNs) model is a functional abstraction of the
CHAPER 3 3. Introduction An artificial neural networs (ANNs) model is a functional abstraction of the biological neural structures of the central nervous system. hey are composed of many simple and highly
More informationNeural Networks. Fundamentals Framework for distributed processing Network topologies Training of ANN s Notation Perceptron Back Propagation
Neural Networks Fundamentals Framework for distributed processing Network topologies Training of ANN s Notation Perceptron Back Propagation Neural Networks Historical Perspective A first wave of interest
More informationMachine Learning for LargeScale Data Analysis and Decision Making A. Neural Networks Week #6
Machine Learning for LargeScale Data Analysis and Decision Making 8062917A Neural Networks Week #6 Today Neural Networks A. Modeling B. Fitting C. Deep neural networks Today s material is (adapted)
More informationCS 4700: Foundations of Artificial Intelligence
CS 4700: Foundations of Artificial Intelligence Prof. Bart Selman selman@cs.cornell.edu Machine Learning: Neural Networks R&N 18.7 Intro & perceptron learning 1 2 Neuron: How the brain works # neurons
More informationArtificial Neural Network and Fuzzy Logic
Artificial Neural Network and Fuzzy Logic 1 Syllabus 2 Syllabus 3 Books 1. Artificial Neural Networks by B. Yagnanarayan, PHI  (Cover Topologies part of unit 1 and All part of Unit 2) 2. Neural Networks
More information2015 Todd Neller. A.I.M.A. text figures 1995 Prentice Hall. Used by permission. Neural Networks. Todd W. Neller
2015 Todd Neller. A.I.M.A. text figures 1995 Prentice Hall. Used by permission. Neural Networks Todd W. Neller Machine Learning Learning is such an important part of what we consider "intelligence" that
More informationLast update: October 26, Neural networks. CMSC 421: Section Dana Nau
Last update: October 26, 207 Neural networks CMSC 42: Section 8.7 Dana Nau Outline Applications of neural networks Brains Neural network units Perceptrons Multilayer perceptrons 2 Example Applications
More informationNeural networks. Chapter 20, Section 5 1
Neural networks Chapter 20, Section 5 Chapter 20, Section 5 Outline Brains Neural networks Perceptrons Multilayer perceptrons Applications of neural networks Chapter 20, Section 5 2 Brains 0 neurons of
More informationLecture 4: Feed Forward Neural Networks
Lecture 4: Feed Forward Neural Networks Dr. Roman V Belavkin Middlesex University BIS4435 Biological neurons and the brain A Model of A Single Neuron Neurons as datadriven models Neural Networks Training
More informationChapter 9: The Perceptron
Chapter 9: The Perceptron 9.1 INTRODUCTION At this point in the book, we have completed all of the exercises that we are going to do with the James program. These exercises have shown that distributed
More informationArtificial Neural Networks. Q550: Models in Cognitive Science Lecture 5
Artificial Neural Networks Q550: Models in Cognitive Science Lecture 5 "Intelligence is 10 million rules." Doug Lenat The human brain has about 100 billion neurons. With an estimated average of one thousand
More informationReification of Boolean Logic
526 U1180 neural networks 1 Chapter 1 Reification of Boolean Logic The modern era of neural networks began with the pioneer work of McCulloch and Pitts (1943). McCulloch was a psychiatrist and neuroanatomist;
More informationUsing a Hopfield Network: A Nuts and Bolts Approach
Using a Hopfield Network: A Nuts and Bolts Approach November 4, 2013 Gershon Wolfe, Ph.D. Hopfield Model as Applied to Classification Hopfield network Training the network Updating nodes Sequencing of
More informationPattern Recognition Prof. P. S. Sastry Department of Electronics and Communication Engineering Indian Institute of Science, Bangalore
Pattern Recognition Prof. P. S. Sastry Department of Electronics and Communication Engineering Indian Institute of Science, Bangalore Lecture  27 Multilayer Feedforward Neural networks with Sigmoidal
More informationNeural Networks. CSE 6363 Machine Learning Vassilis Athitsos Computer Science and Engineering Department University of Texas at Arlington
Neural Networks CSE 6363 Machine Learning Vassilis Athitsos Computer Science and Engineering Department University of Texas at Arlington 1 Perceptrons x 0 = 1 x 1 x 2 z = h w T x Output: z x D A perceptron
More informationLab 5: 16 th April Exercises on Neural Networks
Lab 5: 16 th April 01 Exercises on Neural Networks 1. What are the values of weights w 0, w 1, and w for the perceptron whose decision surface is illustrated in the figure? Assume the surface crosses the
More informationNeural Networks biological neuron artificial neuron 1
Neural Networks biological neuron artificial neuron 1 A twolayer neural network Output layer (activation represents classification) Weighted connections Hidden layer ( internal representation ) Input
More informationNeural Networks (Part 1) Goals for the lecture
Neural Networks (Part ) Mark Craven and David Page Computer Sciences 760 Spring 208 www.biostat.wisc.edu/~craven/cs760/ Some of the slides in these lectures have been adapted/borrowed from materials developed
More informationMultilayer Perceptrons and Backpropagation
Multilayer Perceptrons and Backpropagation Informatics 1 CG: Lecture 7 Chris Lucas School of Informatics University of Edinburgh January 31, 2017 (Slides adapted from Mirella Lapata s.) 1 / 33 Reading:
More informationNeural Networks, Computation Graphs. CMSC 470 Marine Carpuat
Neural Networks, Computation Graphs CMSC 470 Marine Carpuat Binary Classification with a Multilayer Perceptron φ A = 1 φ site = 1 φ located = 1 φ Maizuru = 1 φ, = 2 φ in = 1 φ Kyoto = 1 φ priest = 0 φ
More informationMachine Learning and Data Mining. Multilayer Perceptrons & Neural Networks: Basics. Prof. Alexander Ihler
+ Machine Learning and Data Mining Multilayer Perceptrons & Neural Networks: Basics Prof. Alexander Ihler Linear Classifiers (Perceptrons) Linear Classifiers a linear classifier is a mapping which partitions
More informationSections 18.6 and 18.7 Analysis of Artificial Neural Networks
Sections 18.6 and 18.7 Analysis of Artificial Neural Networks CS4811  Artificial Intelligence Nilufer Onder Department of Computer Science Michigan Technological University Outline Univariate regression
More informationUnit III. A Survey of Neural Network Model
Unit III A Survey of Neural Network Model 1 Single Layer Perceptron Perceptron the first adaptive network architecture was invented by Frank Rosenblatt in 1957. It can be used for the classification of
More information) (d o f. For the previous layer in a neural network (just the rightmost layer if a single neuron), the required update equation is: 2.
1 Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.034 Artificial Intelligence, Fall 2011 Recitation 8, November 3 Corrected Version & (most) solutions
More informationArtificial Neural Network : Training
Artificial Neural Networ : Training Debasis Samanta IIT Kharagpur debasis.samanta.iitgp@gmail.com 06.04.2018 Debasis Samanta (IIT Kharagpur) Soft Computing Applications 06.04.2018 1 / 49 Learning of neural
More informationAnalysis of Multilayer Neural Network Modeling and Long ShortTerm Memory
Analysis of Multilayer Neural Network Modeling and Long ShortTerm Memory Danilo López, Nelson Vera, Luis Pedraza International Science Index, Mathematical and Computational Sciences waset.org/publication/10006216
More informationArtificial Neural Networks Examination, June 2005
Artificial Neural Networks Examination, June 2005 Instructions There are SIXTY questions. (The pass mark is 30 out of 60). For each question, please select a maximum of ONE of the given answers (either
More informationLecture 5: Logistic Regression. Neural Networks
Lecture 5: Logistic Regression. Neural Networks Logistic regression Comparison with generative models Feedforward neural networks Backpropagation Tricks for training neural networks COMP652, Lecture
More informationCh.8 Neural Networks
Ch.8 Neural Networks Hantao Zhang http://www.cs.uiowa.edu/ hzhang/c145 The University of Iowa Department of Computer Science Artificial Intelligence p.1/?? Brains as Computational Devices Motivation: Algorithms
More informationSingle layer NN. Neuron Model
Single layer NN We consider the simple architecture consisting of just one neuron. Generalization to a single layer with more neurons as illustrated below is easy because: M M The output units are independent
More informationNeural Networks for Machine Learning. Lecture 2a An overview of the main types of neural network architecture
Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network architecture Geoffrey Hinton with Nitish Srivastava Kevin Swersky Feedforward neural networks These are
More informationECE 471/571  Lecture 17. Types of NN. History. Back Propagation. Recurrent (feedback during operation) Feedforward
ECE 47/57  Lecture 7 Back Propagation Types of NN Recurrent (feedback during operation) n Hopfield n Kohonen n Associative memory Feedforward n No feedback during operation or testing (only during determination
More informationMachine Learning. Neural Networks. (slides from Domingos, Pardo, others)
Machine Learning Neural Networks (slides from Domingos, Pardo, others) For this week, Reading Chapter 4: Neural Networks (Mitchell, 1997) See Canvas For subsequent weeks: Scaling Learning Algorithms toward
More informationArtificial Neural Networks
Introduction ANN in Action Final Observations Application: Poverty Detection Artificial Neural Networks Alvaro J. Riascos Villegas University of los Andes and Quantil July 6 2018 Artificial Neural Networks
More informationEEE 241: Linear Systems
EEE 4: Linear Systems Summary # 3: Introduction to artificial neural networks DISTRIBUTED REPRESENTATION An ANN consists of simple processing units communicating with each other. The basic elements of
More informationSections 18.6 and 18.7 Artificial Neural Networks
Sections 18.6 and 18.7 Artificial Neural Networks CS4811  Artificial Intelligence Nilufer Onder Department of Computer Science Michigan Technological University Outline The brain vs. artifical neural
More informationECLT 5810 Classification Neural Networks. Reference: Data Mining: Concepts and Techniques By J. Hand, M. Kamber, and J. Pei, Morgan Kaufmann
ECLT 5810 Classification Neural Networks Reference: Data Mining: Concepts and Techniques By J. Hand, M. Kamber, and J. Pei, Morgan Kaufmann Neural Networks A neural network is a set of connected input/output
More informationMaster Recherche IAC TC2: Apprentissage Statistique & Optimisation
Master Recherche IAC TC2: Apprentissage Statistique & Optimisation Alexandre Allauzen Anne Auger Michèle Sebag LIMSI LRI Oct. 4th, 2012 This course Bioinspired algorithms Classical Neural Nets History
More informationSections 18.6 and 18.7 Artificial Neural Networks
Sections 18.6 and 18.7 Artificial Neural Networks CS4811  Artificial Intelligence Nilufer Onder Department of Computer Science Michigan Technological University Outline The brain vs artifical neural networks
More informationAdvanced statistical methods for data analysis Lecture 2
Advanced statistical methods for data analysis Lecture 2 RHUL Physics www.pp.rhul.ac.uk/~cowan Universität Mainz Klausurtagung des GK Eichtheorien exp. Tests... Bullay/Mosel 15 17 September, 2008 1 Outline
More information4. Multilayer Perceptrons
4. Multilayer Perceptrons This is a supervised errorcorrection learning algorithm. 1 4.1 Introduction A multilayer feedforward network consists of an input layer, one or more hidden layers, and an output
More informationArtificial Neural Networks
Artificial Neural Networks Oliver Schulte  CMPT 310 Neural Networks Neural networks arise from attempts to model human/animal brains Many models, many claims of biological plausibility We will focus on
More informationNeural Networks Learning the network: Backprop , Fall 2018 Lecture 4
Neural Networks Learning the network: Backprop 11785, Fall 2018 Lecture 4 1 Recap: The MLP can represent any function The MLP can be constructed to represent anything But how do we construct it? 2 Recap:
More informationLinear classification with logistic regression
Section 8.6. Regression and Classification with Linear Models 725 Proportion correct.9.7 Proportion correct.9.7 2 3 4 5 6 7 2 4 6 8 2 4 6 8 Number of weight updates Number of weight updates Number of weight
More informationArtificial Neural Networks. MGS Lecture 2
Artificial Neural Networks MGS 2018  Lecture 2 OVERVIEW Biological Neural Networks Cell Topology: Input, Output, and Hidden Layers Functional description Cost functions Training ANNs BackPropagation
More informationArtificial Neural Networks Examination, March 2004
Artificial Neural Networks Examination, March 2004 Instructions There are SIXTY questions (worth up to 60 marks). The exam mark (maximum 60) will be added to the mark obtained in the laborations (maximum
More informationCS 4700: Foundations of Artificial Intelligence
CS 4700: Foundations of Artificial Intelligence Prof. Bart Selman selman@cs.cornell.edu Machine Learning: Neural Networks R&N 18.7 Intro & perceptron learning 1 2 Neuron: How the brain works # neurons
More informationLecture 6. Notes on Linear Algebra. Perceptron
Lecture 6. Notes on Linear Algebra. Perceptron COMP90051 Statistical Machine Learning Semester 2, 2017 Lecturer: Andrey Kan Copyright: University of Melbourne This lecture Notes on linear algebra Vectors
More informationArtificial Intelligence (AI) Common AI Methods. Training. Signals to Perceptrons. Artificial Neural Networks (ANN) Artificial Intelligence
Artificial Intelligence (AI) Artificial Intelligence AI is an attempt to reproduce intelligent reasoning using machines * * H. M. Cartwright, Applications of Artificial Intelligence in Chemistry, 1993,
More informationArtificial Neural Networks. Historical description
Artificial Neural Networks Historical description Victor G. Lopez 1 / 23 Artificial Neural Networks (ANN) An artificial neural network is a computational model that attempts to emulate the functions of
More informationUnit 8: Introduction to neural networks. Perceptrons
Unit 8: Introduction to neural networks. Perceptrons D. Balbontín Noval F. J. Martín Mateos J. L. Ruiz Reina A. Riscos Núñez Departamento de Ciencias de la Computación e Inteligencia Artificial Universidad
More informationArtificial Neural Network
Artificial Neural Network Eung Je Woo Department of Biomedical Engineering Impedance Imaging Research Center (IIRC) Kyung Hee University Korea ejwoo@khu.ac.kr Neuron and Neuron Model McCulloch and Pitts
More informationNeural Networks and Deep Learning
Neural Networks and Deep Learning Professor Ameet Talwalkar November 12, 2015 Professor Ameet Talwalkar Neural Networks and Deep Learning November 12, 2015 1 / 16 Outline 1 Review of last lecture AdaBoost
More informationComputational statistics
Computational statistics Lecture 3: Neural networks Thierry Denœux 5 March, 2016 Neural networks A class of learning methods that was developed separately in different fields statistics and artificial
More informationMining Classification Knowledge
Mining Classification Knowledge Remarks on NonSymbolic Methods JERZY STEFANOWSKI Institute of Computing Sciences, Poznań University of Technology SE lecture revision 2013 Outline 1. Bayesian classification
More informationLearning and Memory in Neural Networks
Learning and Memory in Neural Networks Guy Billings, Neuroinformatics Doctoral Training Centre, The School of Informatics, The University of Edinburgh, UK. Neural networks consist of computational units
More information