Introduction to Artificial Neural Networks
|
|
- Geoffrey Williamson
- 6 years ago
- Views:
Transcription
1 Facultés Universitaires Notre-Dame de la Paix 27 March 2007
2 Outline 1 Introduction 2 Fundamentals Biological neuron Artificial neuron Artificial Neural Network
3 Outline 3 Single-layer ANN Perceptron Adaline Limitations 4 Topology Generalised Delta Rule Deficiencies 5 Jordan Network Hopfield Network
4 Outline 6 Application Smart sweepers 7 Conclusion
5 Introduction Fundamentals 1 Introduction 2 Fundamentals Biological neuron Artificial neuron Artificial Neural Network
6 History Introduction Fundamentals 1943 : W. McCulloch & W. Pitts : first model of artificial neuron 1949 : D. Hebb describes the first learning rule (Hebb s Law) 1957 : F. Rosenblatt designs the Perceptron 1965 : Nils J. Nilsson publishes Learning Machines (automated learning fundamentals)
7 History Introduction Fundamentals 1969 : M. Minsky & S. Papert expose limitations of Perceptron (XOR problem) 1975 : First multi-layer ANN with training algorithm (Cognitron) 1982 : Hopfield networks (J. Hopfield), Self-Organizing Map (T. Kohonen) 1986 : Backpropagation algorithm (R. Williams, D. Rumelhart & G. Hinton)
8 Type of problems Introduction Fundamentals ANN can be used to solve certain types of problems : Classification Pattern recognition Artificial Intelligence Unknown function approximation Model complicated functions Stock exchange estimations Data processing (filtering, clustering,...)...
9 Introduction Fundamentals Biological neuron Artificial neuron Artificial Neural Network 1 Introduction 2 Fundamentals Biological neuron Artificial neuron Artificial Neural Network
10 Introduction Fundamentals A view of the biological neuron Biological neuron Artificial neuron Artificial Neural Network The human nervous system is composed of about neurons. The synapses are the connections between axon terminals and dendrites. The synapses are characterized by a level of effectiveness.
11 Introduction Fundamentals A view of the biological neuron Biological neuron Artificial neuron Artificial Neural Network The impulses received at each dendrites are summed together. If the sum is above the stimulation threshold, the nucleus emits a spike down the axon.
12 Introduction Fundamentals A view of the artificial neuron Biological neuron Artificial neuron Artificial Neural Network x i : input from unit i w ij : weight of the connection from unit i to j θ j : bias of unit j ϕ : activation function o j : state of activation of unit j
13 Propagation rule Introduction Fundamentals Biological neuron Artificial neuron Artificial Neural Network Sigma unit type : net j = n w ij x i i=1 Sigma-pi unit type (Feldman & Ballard) : net j = n m w ij x ik i=1 k=1
14 Activation function Introduction Fundamentals Biological neuron Artificial neuron Artificial Neural Network Threshold function (Heaviside or sgn) : { 1 if v 0 ϕ(v) = 0 if v < 0 Semi-linear function : 1 if v 1 2 ϕ(v) = v if 1 2 < v < if v 1 2 Sigmoid function : ϕ(v) = e kv
15 Topology Introduction Fundamentals Biological neuron Artificial neuron Artificial Neural Network Separation can be made between two types of ANN structure : Feed-forward : acyclic and layer-decomposable graph (Perceptron, Adaline) Recurrent (Jordan, Hopfield, Kohonen)
16 What is an ANN? Introduction Fundamentals Biological neuron Artificial neuron Artificial Neural Network ANN : I O I : set of inputs O : set of outputs Initially, the ANN will not respond correctly to given inputs. { } weights Because are not adapted. biases
17 Introduction Fundamentals How does an ANN learns? Biological neuron Artificial neuron Artificial Neural Network Most interesting characteristic of ANN is the capacity to generalize information from samples. This generalisation occurs through the learning process. Main idea : successive weight adjustment (gradient descent method) Supervised learning (or Associative learning) Unsupervised learning (or Self-organisation)
18 Supervised learning Introduction Fundamentals Biological neuron Artificial neuron Artificial Neural Network Idea : generate a population of input-output pairs feed ANN with input re-adjust weights and biases if the ANN doesn t output what is expected Representation of data is imposed to the ANN
19 Unsupervised learning Introduction Fundamentals Biological neuron Artificial neuron Artificial Neural Network Idea : generate a population of input pairs feed the population to the ANN let it extract statistical properties from the population Representation of data is defined by the ANN
20 Introduction Fundamentals Hebbian Learning Rule Biological neuron Artificial neuron Artificial Neural Network w ij = w ij + δw ij δw ij = γo i o j with : w ij : weight from unit i to unit j γ : learning rate o i : state of activation of unit i Virtually, all learning rules can be considered as variants of HLR.
21 Learning Rate Introduction Fundamentals Biological neuron Artificial neuron Artificial Neural Network Defines the speed at which the ANN will learn Usually a constant 0 < γ 1 γ 0 : slow convergence but stable solution. γ 1 : fast convergence but instable solution.
22 Over-fitting Introduction Fundamentals Biological neuron Artificial neuron Artificial Neural Network Excessive learning or inadapted training set can lead to an over-fitted network. An over-fitted ANN is specialised for the set which was used to train it. It lost a great part of its generalisation capability.
23 Two issues Introduction Fundamentals Biological neuron Artificial neuron Artificial Neural Network Representational power : ability of an ANN to represent a desired function. Since an ANN is built from standard functions, it can only approximate the desired function, even for an optimal set of weights. Ergo, the approximation error can never be equal to 0. Learning algorithm : given there exists a set of optimal weights (i.e. which minimize the approximation error), is there a procedure to compute them?
24 Single-layer ANN Perceptron Adaline Limitations 3 Single-layer ANN Perceptron Adaline Limitations 4 Topology Generalised Delta Rule Deficiencies 5 Jordan Network Hopfield Network
25 Perceptron Single-layer ANN Perceptron Adaline Limitations Proposed by F. Rosenblatt in A Perceptron is a single-layer ANN. Composed of one or more output neurons, connected to all inputs. Typically used as a linear classifier.
26 Simple case Single-layer ANN Perceptron Adaline Limitations Consider the following Perceptron : 1 neuron 2 inputs 1 output Threshold-type activation function : { 1 if v > 0 ϕ(v) = 1 otherwise We can use it as a classifier with a separation line : w 1 x 1 + w 2 x 2 + θ = 0
27 Simple case Single-layer ANN Perceptron Adaline Limitations x 2 = w 1 w 2 x 1 θ w 2
28 Perceptron learning Single-layer ANN Perceptron Adaline Limitations Learning consists of a successive weight adjustment : w ij = w ij + w ij θ j = θ j + θ j Problem : how to compute the w ij and θ j?
29 Single-layer ANN Perceptron Learning Rule Perceptron Adaline Limitations Consider a set of learning samples (x, d(x)), with : x : input vector d(x) : desired output Learning Method : 1. Start with random weights for the connections; 2. Select an input vector x from the set of training samples; 3. If o d(x) (the perceptron gives an incorrect response), modify all connections w ij according to : w ij = d j (x)x i θ j = d j (x) 4. Go back to 2.
30 Single-layer ANN Convergence Theorem Perceptron Adaline Limitations Theorem 1. If there exists a set of connection weights w which is able to perform the transformation o = d(x), the perceptron learning rule will converge to some solution (which may or may not be the same as w ) in a finite number of steps for any initial choice of the weights.
31 Numerical example Single-layer ANN Perceptron Adaline Limitations Initial parameters : w 1 = 1 w 2 = 2 θ = 2 Set of samples : Sample A : x = (0.5, 1.5) ; d(x) = 1 Sample B : x = (-0.5, 0.5) ; d(x) = -1 Sample C : x = (0.5, 0.5) ; d(x) = 1
32 Single-layer ANN Numerical example (cont d) Perceptron Adaline Limitations Sample A : Sample B : net = = 1.5 > 0 o = 1 net = = o = 1
33 Single-layer ANN Numerical example (cont d) Perceptron Adaline Limitations Sample C : Updated weights and bias : net = = o = 1 w 1 = w 1 + w 1 = = 1.5; w 2 = w 1 + w 2 = = 2.5; θ = θ + θ = = 1
34 Single-layer ANN Numerical example (cont d) Perceptron Adaline Limitations
35 Single-layer ANN Adaptive Linear Element Perceptron Adaline Limitations Proposed by B. Widrow and T. Hoff in Use a generalised version of the PLR, known as the Delta Rule. Focus is put on net j instead of o j.
36 Single-layer ANN Delta rule (Widrow-Hoff) Perceptron Adaline Limitations Main idea : minimize the error in the output through gradient descent. w i = γ(d p y p )x i with γ : learning rate d p : expected output for input p y p : obtained output for input p
37 Single-layer ANN Delta Rule derivation Perceptron Adaline Limitations Consider a single-layer ANN with an output unit using a linear activation function ; y = w i x i + θ i The objective is to minimize the total error given by : E = 1 (d p y p ) 2 2 p The idea is to adjust the weight proportionately to the negative of the derivative of the error with respect to each weight : p w i = γ Ep w i
38 Single-layer ANN Delta Rule derivation (cont d) Perceptron Adaline Limitations We can split the right derivative following the chain rule : E p w i = Ep y p y p w i The right derivative can be rewritten as : y p w i = x i because of the linearity of the activation function.
39 Single-layer ANN Delta Rule derivation (cont d) Perceptron Adaline Limitations The left derivative can be rewritten as : We obtain the Delta Rule : E p y p = (dp y p ) p w i = γ(d p y p )x i
40 XOR Problem Single-layer ANN Perceptron Adaline Limitations If no linear separation exists, single-layer ANN cannot classify properly. This limitation was exposed by Minsky and Papert through the XOR Problem. It is impossible to teach a single-layer ANN to solve the XOR Problem. Solution : add hidden layers to the ANN.
41 Single-layer ANN Topology Generalised Delta Rule Deficiencies 3 Single-layer ANN Perceptron Adaline Limitations 4 Topology Generalised Delta Rule Deficiencies 5 Jordan Network Hopfield Network
42 Topology Single-layer ANN Topology Generalised Delta Rule Deficiencies A multi-layer ANN is composed of : an input layer one or more hidden layer(s) an output layer In most applications, a single hidden layer is used with sigmoid activation functions.
43 Topology Single-layer ANN Topology Generalised Delta Rule Deficiencies
44 Single-layer ANN Generalised Delta Rule Topology Generalised Delta Rule Deficiencies An important assumption made for the Delta Rule was the linearity of the activation function. In a multi-layer ANN, this assumption no longer holds. We must find a way to generalize the Delta Rule, so that it doesn t restrain the weight adaptation to the output layer.
45 Derivation Single-layer ANN Topology Generalised Delta Rule Deficiencies Consider units with non-linear activation function : y p k = ϕ(netp k ) where net p k = i w ik y p i + θ k
46 Derivation (cont d) Single-layer ANN Topology Generalised Delta Rule Deficiencies The modification we should apply to each weight is given by : p w ik = γ Ep w ik In which E p, the total error is defined by : E p = 1 2 N o o=1 By using the chain rule we obtain : (d p o y p o) 2 p w ik = γ Ep net p net p k k w ik
47 Derivation (cont d) Single-layer ANN Topology Generalised Delta Rule Deficiencies The right derivative can be rewritten as : net p k w ik = y p i If we define δ p k = Ep net p k we obtain an update rule which is similar to the Delta Rule : p w ik = γδ p k yp i The problem now is to define this δ p k network. for the different unit k in the
48 Derivation (cont d) Single-layer ANN Topology Generalised Delta Rule Deficiencies By using the chain rule, we rewrite δ p k : δ p k = Ep net p k = Ep y p k The right derivative can be rewritten as : y p k net p k y p k net p k = ϕ (net p k ) For the left derivative, we must consider two cases : k is an output unit o ; δ p o = (d p o y p o)ϕ (net p o)
49 Derivation (cont d) Single-layer ANN Topology Generalised Delta Rule Deficiencies k is a hidden unit h ; E p y p h N o = δow p ho o=1 We can use this to write : N o δ p h = ϕ (net p h ) δow p ho o=1
50 Derivation (cont d) Single-layer ANN Topology Generalised Delta Rule Deficiencies The two equations : δ p o = (d p o y p o)ϕ (net p o) (1) N o δ p h = ϕ (net p h ) δow p ho (2) Define a recursive procedure which can be used to adjust the weights of the network. It constitutes the Generalised Delta Rule for a feed-forward network of non-linear units. o=1
51 Single-layer ANN Learning rate and Momentum Topology Generalised Delta Rule Deficiencies In order to have fast convergence with a stable solution, a momentum term is added to the variation of the weight. w jk (t + 1) = γδ p k yp j + α w jk(t) Instability of the solution is countered because the change in the weights is dependant of the previous change. It is possible to increase the learning rate γ without causing oscillation in the solution.
52 Single-layer ANN Learning rate and Momentum Topology Generalised Delta Rule Deficiencies a) γ 0 b) γ 1 c) γ 1 with a momentum term added
53 Deficiencies Single-layer ANN Topology Generalised Delta Rule Deficiencies Network paralysis : as the network is trained, the weights can increase to very high values (either positive or negative), so does net j. Because of the sigmoid function, the activation will be very close to zero or very close to one. In that case, the back-propagation algorithm may come to a standstill. Local Minima : because of the shape of the error function for a complex network, the gradient method can find itself trapped in a local minima. Some methods (probabilistic) can avoid this problem but are very slow. It is also possible to increase the number of hidden units without going beyond a certain threshold.
54 Single-layer ANN Jordan Network Hopfield Network 3 Single-layer ANN Perceptron Adaline Limitations 4 Topology Generalised Delta Rule Deficiencies 5 Jordan Network Hopfield Network
55 Jordan Network Single-layer ANN Jordan Network Hopfield Network Proposed by Jordan in Activation values of the output units are fed back into the input layer through so-called state units. The weights from output units to these state units is fixed to +1. Thus, the learning rules which apply to multi-layer ANN can be used to train Jordan Networks.
56 Jordan Network Single-layer ANN Jordan Network Hopfield Network
57 Hopfield Network Single-layer ANN Jordan Network Hopfield Network Proposed by J. Hopfield in Consists of a fully-interconnected network of N neurons which are both input and output. Updates are made asynchronously and independently. Activation values are binary (+1 / -1). Can be used as an associative memory or for optimisation problems (salesman problem).
58 Single-layer ANN Hopfield Auto-Associator Jordan Network Hopfield Network
59 Application Conclusion Smart sweepers 6 Application Smart sweepers 7 Conclusion
60 Smart sweepers Application Conclusion Smart sweepers Objective : train minesweepers to pick up mines in a 2-dimensionnal field. Parameters of the network : Topology : Feed-forward multi-layer ANN. 4 input units 6 hidden units on one layer 2 output units Activation function : sigmoid function. Learning rule : genetic algorithm.
61 Smart sweepers Application Conclusion Smart sweepers Input is composed of two vectors : Vector defining the direction of the closest mine. Vector defining the direction towards which the minesweeper is pointing. Output is composed of two components, the speeds of left and right track.
62 Smart sweepers Application Conclusion Smart sweepers Each minesweeper has its own set of weights. The ANN works for a certain amount of time T. During this period of time, each mine found increases the fitness of the sweeper. Afterwards, the GA starts running to create the new generation of weight sets.
63 Smart sweepers Application Conclusion Smart sweepers
64 Smart sweepers Application Conclusion Smart sweepers
65 Conclusion Application Conclusion Single-layer ANN : Limited representationnal power : restricted to linear classifier Linearity of the system convergence to the optimal solution (optimal weight vector) : Unlimited representationnal power : can model non-linear problems Non-linearity doesn t guarantee the convergence to an optimal solution
66 Conclusion Application Conclusion Choice of a representative learning sample is essential to obtain the expected behavior and to avoid over-fitting. Combination with other approaches can prove to be effective (genetic algorithms for instance)
67 References Application Conclusion B. Kröse, P. van der Smagt An introduction to Neural Networks, Eighth edition. S. Singh Neural Network Recognition of Hand-printed Characters.
Introduction to Neural Networks
Introduction to Neural Networks What are (Artificial) Neural Networks? Models of the brain and nervous system Highly parallel Process information much more like the brain than a serial computer Learning
More informationNeural Networks. Fundamentals Framework for distributed processing Network topologies Training of ANN s Notation Perceptron Back Propagation
Neural Networks Fundamentals Framework for distributed processing Network topologies Training of ANN s Notation Perceptron Back Propagation Neural Networks Historical Perspective A first wave of interest
More informationNeural networks. Chapter 20. Chapter 20 1
Neural networks Chapter 20 Chapter 20 1 Outline Brains Neural networks Perceptrons Multilayer networks Applications of neural networks Chapter 20 2 Brains 10 11 neurons of > 20 types, 10 14 synapses, 1ms
More informationLecture 7 Artificial neural networks: Supervised learning
Lecture 7 Artificial neural networks: Supervised learning Introduction, or how the brain works The neuron as a simple computing element The perceptron Multilayer neural networks Accelerated learning in
More informationFeedforward Neural Nets and Backpropagation
Feedforward Neural Nets and Backpropagation Julie Nutini University of British Columbia MLRG September 28 th, 2016 1 / 23 Supervised Learning Roadmap Supervised Learning: Assume that we are given the features
More informationLecture 4: Feed Forward Neural Networks
Lecture 4: Feed Forward Neural Networks Dr. Roman V Belavkin Middlesex University BIS4435 Biological neurons and the brain A Model of A Single Neuron Neurons as data-driven models Neural Networks Training
More informationArtificial Neural Networks. Historical description
Artificial Neural Networks Historical description Victor G. Lopez 1 / 23 Artificial Neural Networks (ANN) An artificial neural network is a computational model that attempts to emulate the functions of
More informationNeural networks. Chapter 19, Sections 1 5 1
Neural networks Chapter 19, Sections 1 5 Chapter 19, Sections 1 5 1 Outline Brains Neural networks Perceptrons Multilayer perceptrons Applications of neural networks Chapter 19, Sections 1 5 2 Brains 10
More informationIntroduction To Artificial Neural Networks
Introduction To Artificial Neural Networks Machine Learning Supervised circle square circle square Unsupervised group these into two categories Supervised Machine Learning Supervised Machine Learning Supervised
More informationSimple Neural Nets For Pattern Classification
CHAPTER 2 Simple Neural Nets For Pattern Classification Neural Networks General Discussion One of the simplest tasks that neural nets can be trained to perform is pattern classification. In pattern classification
More informationNeural Networks. Chapter 18, Section 7. TB Artificial Intelligence. Slides from AIMA 1/ 21
Neural Networks Chapter 8, Section 7 TB Artificial Intelligence Slides from AIMA http://aima.cs.berkeley.edu / 2 Outline Brains Neural networks Perceptrons Multilayer perceptrons Applications of neural
More informationSupervised (BPL) verses Hybrid (RBF) Learning. By: Shahed Shahir
Supervised (BPL) verses Hybrid (RBF) Learning By: Shahed Shahir 1 Outline I. Introduction II. Supervised Learning III. Hybrid Learning IV. BPL Verses RBF V. Supervised verses Hybrid learning VI. Conclusion
More informationLast update: October 26, Neural networks. CMSC 421: Section Dana Nau
Last update: October 26, 207 Neural networks CMSC 42: Section 8.7 Dana Nau Outline Applications of neural networks Brains Neural network units Perceptrons Multilayer perceptrons 2 Example Applications
More informationNeural networks. Chapter 20, Section 5 1
Neural networks Chapter 20, Section 5 Chapter 20, Section 5 Outline Brains Neural networks Perceptrons Multilayer perceptrons Applications of neural networks Chapter 20, Section 5 2 Brains 0 neurons of
More informationNeural Networks and Fuzzy Logic Rajendra Dept.of CSE ASCET
Unit-. Definition Neural network is a massively parallel distributed processing system, made of highly inter-connected neural computing elements that have the ability to learn and thereby acquire knowledge
More informationData Mining Part 5. Prediction
Data Mining Part 5. Prediction 5.5. Spring 2010 Instructor: Dr. Masoud Yaghini Outline How the Brain Works Artificial Neural Networks Simple Computing Elements Feed-Forward Networks Perceptrons (Single-layer,
More informationArtificial Neural Networks
Artificial Neural Networks 鮑興國 Ph.D. National Taiwan University of Science and Technology Outline Perceptrons Gradient descent Multi-layer networks Backpropagation Hidden layer representations Examples
More informationUnit III. A Survey of Neural Network Model
Unit III A Survey of Neural Network Model 1 Single Layer Perceptron Perceptron the first adaptive network architecture was invented by Frank Rosenblatt in 1957. It can be used for the classification of
More informationEEE 241: Linear Systems
EEE 4: Linear Systems Summary # 3: Introduction to artificial neural networks DISTRIBUTED REPRESENTATION An ANN consists of simple processing units communicating with each other. The basic elements of
More informationArtificial Neural Networks The Introduction
Artificial Neural Networks The Introduction 01001110 01100101 01110101 01110010 01101111 01101110 01101111 01110110 01100001 00100000 01110011 01101011 01110101 01110000 01101001 01101110 01100001 00100000
More informationPlan. Perceptron Linear discriminant. Associative memories Hopfield networks Chaotic networks. Multilayer perceptron Backpropagation
Neural Networks Plan Perceptron Linear discriminant Associative memories Hopfield networks Chaotic networks Multilayer perceptron Backpropagation Perceptron Historically, the first neural net Inspired
More informationARTIFICIAL INTELLIGENCE. Artificial Neural Networks
INFOB2KI 2017-2018 Utrecht University The Netherlands ARTIFICIAL INTELLIGENCE Artificial Neural Networks Lecturer: Silja Renooij These slides are part of the INFOB2KI Course Notes available from www.cs.uu.nl/docs/vakken/b2ki/schema.html
More informationArtificial Neural Networks Examination, June 2005
Artificial Neural Networks Examination, June 2005 Instructions There are SIXTY questions. (The pass mark is 30 out of 60). For each question, please select a maximum of ONE of the given answers (either
More informationArtifical Neural Networks
Neural Networks Artifical Neural Networks Neural Networks Biological Neural Networks.................................. Artificial Neural Networks................................... 3 ANN Structure...........................................
More informationPart 8: Neural Networks
METU Informatics Institute Min720 Pattern Classification ith Bio-Medical Applications Part 8: Neural Netors - INTRODUCTION: BIOLOGICAL VS. ARTIFICIAL Biological Neural Netors A Neuron: - A nerve cell as
More informationArtificial Neural Networks Examination, March 2004
Artificial Neural Networks Examination, March 2004 Instructions There are SIXTY questions (worth up to 60 marks). The exam mark (maximum 60) will be added to the mark obtained in the laborations (maximum
More informationLearning and Memory in Neural Networks
Learning and Memory in Neural Networks Guy Billings, Neuroinformatics Doctoral Training Centre, The School of Informatics, The University of Edinburgh, UK. Neural networks consist of computational units
More informationARTIFICIAL NEURAL NETWORK PART I HANIEH BORHANAZAD
ARTIFICIAL NEURAL NETWORK PART I HANIEH BORHANAZAD WHAT IS A NEURAL NETWORK? The simplest definition of a neural network, more properly referred to as an 'artificial' neural network (ANN), is provided
More informationArtificial Neural Networks Examination, June 2004
Artificial Neural Networks Examination, June 2004 Instructions There are SIXTY questions (worth up to 60 marks). The exam mark (maximum 60) will be added to the mark obtained in the laborations (maximum
More informationCS:4420 Artificial Intelligence
CS:4420 Artificial Intelligence Spring 2018 Neural Networks Cesare Tinelli The University of Iowa Copyright 2004 18, Cesare Tinelli and Stuart Russell a a These notes were originally developed by Stuart
More informationIntroduction Biologically Motivated Crude Model Backpropagation
Introduction Biologically Motivated Crude Model Backpropagation 1 McCulloch-Pitts Neurons In 1943 Warren S. McCulloch, a neuroscientist, and Walter Pitts, a logician, published A logical calculus of the
More informationArtificial Neural Networks. Edward Gatt
Artificial Neural Networks Edward Gatt What are Neural Networks? Models of the brain and nervous system Highly parallel Process information much more like the brain than a serial computer Learning Very
More informationSections 18.6 and 18.7 Artificial Neural Networks
Sections 18.6 and 18.7 Artificial Neural Networks CS4811 - Artificial Intelligence Nilufer Onder Department of Computer Science Michigan Technological University Outline The brain vs. artifical neural
More informationSections 18.6 and 18.7 Artificial Neural Networks
Sections 18.6 and 18.7 Artificial Neural Networks CS4811 - Artificial Intelligence Nilufer Onder Department of Computer Science Michigan Technological University Outline The brain vs artifical neural networks
More informationBack-Propagation Algorithm. Perceptron Gradient Descent Multilayered neural network Back-Propagation More on Back-Propagation Examples
Back-Propagation Algorithm Perceptron Gradient Descent Multilayered neural network Back-Propagation More on Back-Propagation Examples 1 Inner-product net =< w, x >= w x cos(θ) net = n i=1 w i x i A measure
More informationLecture 4: Perceptrons and Multilayer Perceptrons
Lecture 4: Perceptrons and Multilayer Perceptrons Cognitive Systems II - Machine Learning SS 2005 Part I: Basic Approaches of Concept Learning Perceptrons, Artificial Neuronal Networks Lecture 4: Perceptrons
More informationUnit 8: Introduction to neural networks. Perceptrons
Unit 8: Introduction to neural networks. Perceptrons D. Balbontín Noval F. J. Martín Mateos J. L. Ruiz Reina A. Riscos Núñez Departamento de Ciencias de la Computación e Inteligencia Artificial Universidad
More informationMachine Learning. Neural Networks. (slides from Domingos, Pardo, others)
Machine Learning Neural Networks (slides from Domingos, Pardo, others) Human Brain Neurons Input-Output Transformation Input Spikes Output Spike Spike (= a brief pulse) (Excitatory Post-Synaptic Potential)
More informationArtificial Neural Network and Fuzzy Logic
Artificial Neural Network and Fuzzy Logic 1 Syllabus 2 Syllabus 3 Books 1. Artificial Neural Networks by B. Yagnanarayan, PHI - (Cover Topologies part of unit 1 and All part of Unit 2) 2. Neural Networks
More informationCOMP9444 Neural Networks and Deep Learning 2. Perceptrons. COMP9444 c Alan Blair, 2017
COMP9444 Neural Networks and Deep Learning 2. Perceptrons COMP9444 17s2 Perceptrons 1 Outline Neurons Biological and Artificial Perceptron Learning Linear Separability Multi-Layer Networks COMP9444 17s2
More informationMachine Learning. Neural Networks. (slides from Domingos, Pardo, others)
Machine Learning Neural Networks (slides from Domingos, Pardo, others) For this week, Reading Chapter 4: Neural Networks (Mitchell, 1997) See Canvas For subsequent weeks: Scaling Learning Algorithms toward
More informationArtificial Neural Networks. Q550: Models in Cognitive Science Lecture 5
Artificial Neural Networks Q550: Models in Cognitive Science Lecture 5 "Intelligence is 10 million rules." --Doug Lenat The human brain has about 100 billion neurons. With an estimated average of one thousand
More informationArtificial Intelligence
Artificial Intelligence Jeff Clune Assistant Professor Evolving Artificial Intelligence Laboratory Announcements Be making progress on your projects! Three Types of Learning Unsupervised Supervised Reinforcement
More informationMachine Learning. Neural Networks. (slides from Domingos, Pardo, others)
Machine Learning Neural Networks (slides from Domingos, Pardo, others) For this week, Reading Chapter 4: Neural Networks (Mitchell, 1997) See Canvas For subsequent weeks: Scaling Learning Algorithms toward
More informationAN INTRODUCTION TO NEURAL NETWORKS. Scott Kuindersma November 12, 2009
AN INTRODUCTION TO NEURAL NETWORKS Scott Kuindersma November 12, 2009 SUPERVISED LEARNING We are given some training data: We must learn a function If y is discrete, we call it classification If it is
More informationMaster Recherche IAC TC2: Apprentissage Statistique & Optimisation
Master Recherche IAC TC2: Apprentissage Statistique & Optimisation Alexandre Allauzen Anne Auger Michèle Sebag LIMSI LRI Oct. 4th, 2012 This course Bio-inspired algorithms Classical Neural Nets History
More informationNeural Networks: Introduction
Neural Networks: Introduction Machine Learning Fall 2017 Based on slides and material from Geoffrey Hinton, Richard Socher, Dan Roth, Yoav Goldberg, Shai Shalev-Shwartz and Shai Ben-David, and others 1
More informationSections 18.6 and 18.7 Analysis of Artificial Neural Networks
Sections 18.6 and 18.7 Analysis of Artificial Neural Networks CS4811 - Artificial Intelligence Nilufer Onder Department of Computer Science Michigan Technological University Outline Univariate regression
More informationMachine Learning. Neural Networks
Machine Learning Neural Networks Bryan Pardo, Northwestern University, Machine Learning EECS 349 Fall 2007 Biological Analogy Bryan Pardo, Northwestern University, Machine Learning EECS 349 Fall 2007 THE
More informationArtificial Neural Networks. Part 2
Artificial Neural Netorks Part Artificial Neuron Model Folloing simplified model of real neurons is also knon as a Threshold Logic Unit x McCullouch-Pitts neuron (943) x x n n Body of neuron f out Biological
More informationArtificial Neural Networks Examination, March 2002
Artificial Neural Networks Examination, March 2002 Instructions There are SIXTY questions (worth up to 60 marks). The exam mark (maximum 60) will be added to the mark obtained in the laborations (maximum
More informationArtificial Neural Networks. MGS Lecture 2
Artificial Neural Networks MGS 2018 - Lecture 2 OVERVIEW Biological Neural Networks Cell Topology: Input, Output, and Hidden Layers Functional description Cost functions Training ANNs Back-Propagation
More informationChapter 2 Single Layer Feedforward Networks
Chapter 2 Single Layer Feedforward Networks By Rosenblatt (1962) Perceptrons For modeling visual perception (retina) A feedforward network of three layers of units: Sensory, Association, and Response Learning
More informationInput layer. Weight matrix [ ] Output layer
MASSACHUSETTS INSTITUTE OF TECHNOLOGY Department of Electrical Engineering and Computer Science 6.034 Artificial Intelligence, Fall 2003 Recitation 10, November 4 th & 5 th 2003 Learning by perceptrons
More informationUsing a Hopfield Network: A Nuts and Bolts Approach
Using a Hopfield Network: A Nuts and Bolts Approach November 4, 2013 Gershon Wolfe, Ph.D. Hopfield Model as Applied to Classification Hopfield network Training the network Updating nodes Sequencing of
More informationAI Programming CS F-20 Neural Networks
AI Programming CS662-2008F-20 Neural Networks David Galles Department of Computer Science University of San Francisco 20-0: Symbolic AI Most of this class has been focused on Symbolic AI Focus or symbols
More informationRevision: Neural Network
Revision: Neural Network Exercise 1 Tell whether each of the following statements is true or false by checking the appropriate box. Statement True False a) A perceptron is guaranteed to perfectly learn
More informationSupervised Learning in Neural Networks
The Norwegian University of Science and Technology (NTNU Trondheim, Norway keithd@idi.ntnu.no March 7, 2011 Supervised Learning Constant feedback from an instructor, indicating not only right/wrong, but
More informationChapter 9: The Perceptron
Chapter 9: The Perceptron 9.1 INTRODUCTION At this point in the book, we have completed all of the exercises that we are going to do with the James program. These exercises have shown that distributed
More informationFundamentals of Neural Networks
Fundamentals of Neural Networks : Soft Computing Course Lecture 7 14, notes, slides www.myreaders.info/, RC Chakraborty, e-mail rcchak@gmail.com, Aug. 10, 2010 http://www.myreaders.info/html/soft_computing.html
More informationCSE 352 (AI) LECTURE NOTES Professor Anita Wasilewska. NEURAL NETWORKS Learning
CSE 352 (AI) LECTURE NOTES Professor Anita Wasilewska NEURAL NETWORKS Learning Neural Networks Classifier Short Presentation INPUT: classification data, i.e. it contains an classification (class) attribute.
More informationAdministration. Registration Hw3 is out. Lecture Captioning (Extra-Credit) Scribing lectures. Questions. Due on Thursday 10/6
Administration Registration Hw3 is out Due on Thursday 10/6 Questions Lecture Captioning (Extra-Credit) Look at Piazza for details Scribing lectures With pay; come talk to me/send email. 1 Projects Projects
More informationNeural Networks for Machine Learning. Lecture 2a An overview of the main types of neural network architecture
Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network architecture Geoffrey Hinton with Nitish Srivastava Kevin Swersky Feed-forward neural networks These are
More informationCS 4700: Foundations of Artificial Intelligence
CS 4700: Foundations of Artificial Intelligence Prof. Bart Selman selman@cs.cornell.edu Machine Learning: Neural Networks R&N 18.7 Intro & perceptron learning 1 2 Neuron: How the brain works # neurons
More informationARTIFICIAL NEURAL NETWORKS گروه مطالعاتي 17 بهار 92
ARTIFICIAL NEURAL NETWORKS گروه مطالعاتي 17 بهار 92 BIOLOGICAL INSPIRATIONS Some numbers The human brain contains about 10 billion nerve cells (neurons) Each neuron is connected to the others through 10000
More informationPattern Classification
Pattern Classification All materials in these slides were taen from Pattern Classification (2nd ed) by R. O. Duda,, P. E. Hart and D. G. Stor, John Wiley & Sons, 2000 with the permission of the authors
More informationChapter ML:VI. VI. Neural Networks. Perceptron Learning Gradient Descent Multilayer Perceptron Radial Basis Functions
Chapter ML:VI VI. Neural Networks Perceptron Learning Gradient Descent Multilayer Perceptron Radial asis Functions ML:VI-1 Neural Networks STEIN 2005-2018 The iological Model Simplified model of a neuron:
More informationNeural Nets in PR. Pattern Recognition XII. Michal Haindl. Outline. Neural Nets in PR 2
Neural Nets in PR NM P F Outline Motivation: Pattern Recognition XII human brain study complex cognitive tasks Michal Haindl Faculty of Information Technology, KTI Czech Technical University in Prague
More informationNeural Networks and Deep Learning
Neural Networks and Deep Learning Professor Ameet Talwalkar November 12, 2015 Professor Ameet Talwalkar Neural Networks and Deep Learning November 12, 2015 1 / 16 Outline 1 Review of last lecture AdaBoost
More informationCMSC 421: Neural Computation. Applications of Neural Networks
CMSC 42: Neural Computation definition synonyms neural networks artificial neural networks neural modeling connectionist models parallel distributed processing AI perspective Applications of Neural Networks
More informationNeural Networks (Part 1) Goals for the lecture
Neural Networks (Part ) Mark Craven and David Page Computer Sciences 760 Spring 208 www.biostat.wisc.edu/~craven/cs760/ Some of the slides in these lectures have been adapted/borrowed from materials developed
More informationNeural Networks DWML, /25
DWML, 2007 /25 Neural networks: Biological and artificial Consider humans: Neuron switching time 0.00 second Number of neurons 0 0 Connections per neuron 0 4-0 5 Scene recognition time 0. sec 00 inference
More informationSimulating Neural Networks. Lawrence Ward P465A
Simulating Neural Networks Lawrence Ward P465A 1. Neural network (or Parallel Distributed Processing, PDP) models are used for a wide variety of roles, including recognizing patterns, implementing logic,
More informationNeural Networks Lecturer: J. Matas Authors: J. Matas, B. Flach, O. Drbohlav
Neural Networks 30.11.2015 Lecturer: J. Matas Authors: J. Matas, B. Flach, O. Drbohlav 1 Talk Outline Perceptron Combining neurons to a network Neural network, processing input to an output Learning Cost
More informationMultilayer Neural Networks
Multilayer Neural Networks Multilayer Neural Networks Discriminant function flexibility NON-Linear But with sets of linear parameters at each layer Provably general function approximators for sufficient
More informationPracticals 5 : Perceptron
Université Paul Sabatier M2 SE Data Mining Practicals 5 : Perceptron Framework The aim of this last session is to introduce the basics of neural networks theory through the special case of the perceptron.
More informationAn artificial neural networks (ANNs) model is a functional abstraction of the
CHAPER 3 3. Introduction An artificial neural networs (ANNs) model is a functional abstraction of the biological neural structures of the central nervous system. hey are composed of many simple and highly
More informationArtificial neural networks
Artificial neural networks Chapter 8, Section 7 Artificial Intelligence, spring 203, Peter Ljunglöf; based on AIMA Slides c Stuart Russel and Peter Norvig, 2004 Chapter 8, Section 7 Outline Brains Neural
More informationArtificial Neural Network
Artificial Neural Network Contents 2 What is ANN? Biological Neuron Structure of Neuron Types of Neuron Models of Neuron Analogy with human NN Perceptron OCR Multilayer Neural Network Back propagation
More informationIntroduction to Natural Computation. Lecture 9. Multilayer Perceptrons and Backpropagation. Peter Lewis
Introduction to Natural Computation Lecture 9 Multilayer Perceptrons and Backpropagation Peter Lewis 1 / 25 Overview of the Lecture Why multilayer perceptrons? Some applications of multilayer perceptrons.
More informationGrundlagen der Künstlichen Intelligenz
Grundlagen der Künstlichen Intelligenz Neural networks Daniel Hennes 21.01.2018 (WS 2017/18) University Stuttgart - IPVS - Machine Learning & Robotics 1 Today Logistic regression Neural networks Perceptron
More informationComputational Intelligence Winter Term 2017/18
Computational Intelligence Winter Term 207/8 Prof. Dr. Günter Rudolph Lehrstuhl für Algorithm Engineering (LS ) Fakultät für Informatik TU Dortmund Plan for Today Single-Layer Perceptron Accelerated Learning
More informationNeural Networks biological neuron artificial neuron 1
Neural Networks biological neuron artificial neuron 1 A two-layer neural network Output layer (activation represents classification) Weighted connections Hidden layer ( internal representation ) Input
More informationIntroduction to Neural Networks
Introduction to Neural Networks Vincent Barra LIMOS, UMR CNRS 6158, Blaise Pascal University, Clermont-Ferrand, FRANCE January 4, 2011 1 / 46 1 INTRODUCTION Introduction History Brain vs. ANN Biological
More informationThe perceptron learning algorithm is one of the first procedures proposed for learning in neural network models and is mostly credited to Rosenblatt.
1 The perceptron learning algorithm is one of the first procedures proposed for learning in neural network models and is mostly credited to Rosenblatt. The algorithm applies only to single layer models
More information4. Multilayer Perceptrons
4. Multilayer Perceptrons This is a supervised error-correction learning algorithm. 1 4.1 Introduction A multilayer feedforward network consists of an input layer, one or more hidden layers, and an output
More informationCourse 395: Machine Learning - Lectures
Course 395: Machine Learning - Lectures Lecture 1-2: Concept Learning (M. Pantic) Lecture 3-4: Decision Trees & CBC Intro (M. Pantic & S. Petridis) Lecture 5-6: Evaluating Hypotheses (S. Petridis) Lecture
More informationIntroduction to Neural Networks
Introduction to Neural Networks Pr. Fabien MOUTARDE Center for Robotics MINES ParisTech PSL Université Paris Fabien.Moutarde@mines-paristech.fr http://people.mines-paristech.fr/fabien.moutarde Introduction
More information2015 Todd Neller. A.I.M.A. text figures 1995 Prentice Hall. Used by permission. Neural Networks. Todd W. Neller
2015 Todd Neller. A.I.M.A. text figures 1995 Prentice Hall. Used by permission. Neural Networks Todd W. Neller Machine Learning Learning is such an important part of what we consider "intelligence" that
More informationIntroduction and Perceptron Learning
Artificial Neural Networks Introduction and Perceptron Learning CPSC 565 Winter 2003 Christian Jacob Department of Computer Science University of Calgary Canada CPSC 565 - Winter 2003 - Emergent Computing
More informationNeural Networks. Mark van Rossum. January 15, School of Informatics, University of Edinburgh 1 / 28
1 / 28 Neural Networks Mark van Rossum School of Informatics, University of Edinburgh January 15, 2018 2 / 28 Goals: Understand how (recurrent) networks behave Find a way to teach networks to do a certain
More informationNeural Networks with Applications to Vision and Language. Feedforward Networks. Marco Kuhlmann
Neural Networks with Applications to Vision and Language Feedforward Networks Marco Kuhlmann Feedforward networks Linear separability x 2 x 2 0 1 0 1 0 0 x 1 1 0 x 1 linearly separable not linearly separable
More informationComputational Intelligence
Plan for Today Single-Layer Perceptron Computational Intelligence Winter Term 00/ Prof. Dr. Günter Rudolph Lehrstuhl für Algorithm Engineering (LS ) Fakultät für Informatik TU Dortmund Accelerated Learning
More informationArtificial Neural Networks (ANN) Xiaogang Su, Ph.D. Department of Mathematical Science University of Texas at El Paso
Artificial Neural Networks (ANN) Xiaogang Su, Ph.D. Department of Mathematical Science University of Texas at El Paso xsu@utep.edu Fall, 2018 Outline Introduction A Brief History ANN Architecture Terminology
More information22c145-Fall 01: Neural Networks. Neural Networks. Readings: Chapter 19 of Russell & Norvig. Cesare Tinelli 1
Neural Networks Readings: Chapter 19 of Russell & Norvig. Cesare Tinelli 1 Brains as Computational Devices Brains advantages with respect to digital computers: Massively parallel Fault-tolerant Reliable
More information(Feed-Forward) Neural Networks Dr. Hajira Jabeen, Prof. Jens Lehmann
(Feed-Forward) Neural Networks 2016-12-06 Dr. Hajira Jabeen, Prof. Jens Lehmann Outline In the previous lectures we have learned about tensors and factorization methods. RESCAL is a bilinear model for
More informationHopfield Neural Network and Associative Memory. Typical Myelinated Vertebrate Motoneuron (Wikipedia) Topic 3 Polymers and Neurons Lecture 5
Hopfield Neural Network and Associative Memory Typical Myelinated Vertebrate Motoneuron (Wikipedia) PHY 411-506 Computational Physics 2 1 Wednesday, March 5 1906 Nobel Prize in Physiology or Medicine.
More informationArtificial Neural Network
Artificial Neural Network Eung Je Woo Department of Biomedical Engineering Impedance Imaging Research Center (IIRC) Kyung Hee University Korea ejwoo@khu.ac.kr Neuron and Neuron Model McCulloch and Pitts
More informationBackpropagation Neural Net
Backpropagation Neural Net As is the case with most neural networks, the aim of Backpropagation is to train the net to achieve a balance between the ability to respond correctly to the input patterns that
More informationEE04 804(B) Soft Computing Ver. 1.2 Class 2. Neural Networks - I Feb 23, Sasidharan Sreedharan
EE04 804(B) Soft Computing Ver. 1.2 Class 2. Neural Networks - I Feb 23, 2012 Sasidharan Sreedharan www.sasidharan.webs.com 3/1/2012 1 Syllabus Artificial Intelligence Systems- Neural Networks, fuzzy logic,
More information