Neural Networks. Prof. Dr. Rudolf Kruse. Computational Intelligence Group Faculty for Computer Science
|
|
- Georgina Fleming
- 6 years ago
- Views:
Transcription
1 Neural Networks Prof. Dr. Rudolf Kruse Computational Intelligence Group Faculty for Computer Science Rudolf Kruse Neural Networks 1
2 Hopfield Networks Rudolf Kruse Neural Networks 2
3 Hopfield Networks A Hopfield network is a neural network with a graph G =(U, C) thatsatisfiesthe following conditions: (i) U hidden =, U in = U out = U, (ii) C = U U {(u, u) u U}. In a Hopfield network all neurons are input as well as output neurons. There are no hidden neurons. Each neuron receives input from all other neurons. Aneuronisnotconnectedtoitself. The connection weights are symmetric, i.e. u, v U, u v : w uv = w vu. Rudolf Kruse Neural Networks 3
4 Hopfield Networks The network input function of each neuron is the weighted sum of the outputs of all other neurons, i.e. u U : f (u) net ( w u, in u )= w u in u = v U {u} w uv out v. The activation function of each neuron is a threshold function, i.e. u U : f (u) act (net u,θ u )= { 1, if netu θ, 1, otherwise. The output function of each neuron is the identity, i.e. u U : f (u) out (act u)=act u. Rudolf Kruse Neural Networks 4
5 Hopfield Networks Alternative activation function u U : f (u) act (net u,θ u, act u )= 1, if net u >θ, 1, if net u <θ, act u, if net u = θ. This activation function has advantages w.r.t. the physical interpretation of a Hopfield network. General weight matrix of a Hopfield network W = 0 w u1 u 2... w u1 u n w u1 u w u2 u n... w u1 u n w u1 u n... 0 Rudolf Kruse Neural Networks 5
6 Hopfield Networks: Examples Very simple Hopfield network u 1 x y 1 W = ( ) x 2 0 y 2 u 2 The behavior of a Hopfield network can depend on the update order. Computations can oscillate if neurons are updated in parallel. Computations always converge if neurons are updated sequentially. Rudolf Kruse Neural Networks 6
7 Hopfield Networks: Examples Parallel update of neuron activations u 1 u 2 input phase 1 1 work phase The computations oscillate, no stable state is reached. Output depends on when the computations are terminated. Rudolf Kruse Neural Networks 7
8 Hopfield Networks: Examples Sequential update of neuron activations u 1 u 2 input phase 1 1 work phase u 1 u 2 input phase 1 1 work phase Regardless of the update order a stable state is reached. Which state is reached depends on the update order. Rudolf Kruse Neural Networks 8
9 Hopfield Networks: Examples Simplified representation of a Hopfield network x 1 0 y 1 x y 2 u u u 2 W = x 3 0 y 3 Symmetric connections between neurons are combined. Inputs and outputs are not explicitely represented. Rudolf Kruse Neural Networks 9
10 Hopfield Networks: State Graph Graph of activation states and transitions +++ u 1 u 2 u 3 u 3 u 2 u 2 u u 3 ++ u 2 u 1 u 3 u 2 u 1 u 3 u 1 u 1 3 u u 1 u 2 u 3 u 1 u 2 u 3 Rudolf Kruse Neural Networks 10
11 Hopfield Networks: Convergence Convergence Theorem: If the activations of the neurons of a Hopfield network are updated sequentially (asynchronously), then a stable state is reached in a finite number of steps. If the neurons are traversed cyclically in an arbitrary, but fixed order, at most n 2 n steps (updates of individual neurons) are needed, where n is the number of neurons of the Hopfield network. The proof is carried out with the help of an energy function. The energy function of a Hopfield network with n neurons u 1,...,u n is E = 1 2 act W act + θ T act = 1 2 u,v U,u v w uv act u act v + u U θ u act u. Rudolf Kruse Neural Networks 11
12 Hopfield Networks: Convergence Consider the energy change resulting from an update that changes an activation: E = E (new) E (old) = ( v U {u} ( = v U {u} ( act (old) u w uv act (new) u w uv act (old) u act (new) u ) ( act v +θ u act (new) u ) act v +θ u act (old) u ) w uv act v v U {u} }{{} =net u net u <θ u :Secondfactorislessthan0. act (new) u = 1 andact (old) u =1,thereforefirstfactorgreaterthan0. Result: E <0. net u θ u :Secondfactorgreaterthanorequalto0. act (new) u =1andact (old) u = 1, therefore first factor less than 0. Result: E 0. θ u ). Rudolf Kruse Neural Networks 12
13 Hopfield Networks: Examples Arrange states in state graph according to their energy E Energy function for example Hopfield network: E = act u1 act u2 2act u1 act u3 act u2 act u3. Rudolf Kruse Neural Networks 13
14 Hopfield Networks: Examples The state graph need not be symmetric 5 E u u u Rudolf Kruse Neural Networks 14
15 Hopfield Networks: Physical Interpretation Physical interpretation: Magnetism AHopfieldnetworkcanbeseenasa(microscopic)modelofmagnetism (so-called Ising model, [Ising 1925]). physical atom magnetic moment (spin) strength of outer magnetic field magnetic coupling of the atoms Hamilton operator of the magnetic field neural neuron activation state threshold value connection weights energy function Rudolf Kruse Neural Networks 15
16 Hopfield Networks: Associative Memory Idea: Use stable states to store patterns First: Store only one pattern x =(act (l) u 1,...,act (l) u n ) { 1, 1} n, n 2, i.e., find weights, so that pattern is a stable state. Necessary and sufficient condition: S(W x θ )= x, where with S :IR n { 1, 1} n, x y i {1,...,n} : y i = { 1, if xi 0, 1, otherwise. Rudolf Kruse Neural Networks 16
17 Hopfield Networks: Associative Memory If θ = 0 anappropriatematrixw can easily be found. It suffices W x = c x with c IR +. Algebraically: Find a matrix W that has a positive eigenvalue w.r.t. x. Choose W = x x T E where x x T is the so-called outer product. With this matrix we have W x = ( x x T ) x }{{} E x = x = n x x = (n 1) x. ( ) = x ( x T x ) x }{{} = x 2 =n Rudolf Kruse Neural Networks 17
18 Hopfield Networks: Associative Memory Hebbian learning rule [Hebb 1949] Written in individual weights the computation of the weight matrix reads: w uv = 0, if u = v, 1, if u v, act (p) u 1, otherwise. Originally derived from a biological analogy. =act (v) u, Strengthen connection between neurons that are active at the same time. Note that this learning rule also stores the complement of the pattern: With W x =(n 1) x it is also W( x )=(n 1)( x ). Rudolf Kruse Neural Networks 18
19 Hopfield Networks: Associative Memory Storing several patterns Choose W x j = m i=1 If patterns are orthogonal, we have and therefore W i x j = x T i x j = = m i=1 m i=1 ( x i x i T ) x j { 0, if i j, n, if i = j, W x j =(n m) x j. m E x j }{{} = x j x i ( x i T x j) m x j Rudolf Kruse Neural Networks 19
20 Hopfield Networks: Associative Memory Storing several patterns Result: As long as m<n, x is a stable state of the Hopfield network. Note that the complements of the patterns are also stored. With W x j =(n m) x j it is also W( x j )=(n m)( x j ). But: Capacity is very small compared to the number of possible states (2 n ). Non-orthogonal patterns: W x j =(n m) x j + m i=1 i j x i ( x T i x j) }{{} disturbance term. Rudolf Kruse Neural Networks 20
21 Associative Memory: Example Example: Store patterns x 1 =(+1, +1, 1, 1) and x 2 =( 1, +1, 1, +1). where W 1 = W = W 1 + W 2 = x 1 x 1 T + x 2 x 2 T 2E , W 2 = The full weight matrix is: Therefore it is W = W x 1 =(+2, +2, 2, 2) and W x 1 =( 2, +2, 2, +2). Rudolf Kruse Neural Networks 21
22 Associative Memory: Examples Example: Storing bit maps of numbers Left: Bit maps stored in a Hopfield network. Right: Reconstruction of a pattern from a random input. Rudolf Kruse Neural Networks 22
23 Hopfield Networks: Associative Memory Training a Hopfield network with the Delta rule Necessary condition for pattern x being a stable state: s(0 + w u1 u 2 act (p) u w u1 u n act (p) u n θ u1 )=act (p) u 1, s(w u2 u 1 act (p) u w u2 u n act (p) u n θ u2 )=act (p) u 2,..... s(w un u 1 act (p) u 1 + w un u 2 act (p) u with the standard threshold function θ un )=act (p) u n. s(x) = { 1, if x 0, 1, otherwise. Rudolf Kruse Neural Networks 23
24 Hopfield Networks: Associative Memory Training a Hopfield network with the Delta rule Turn weight matrix into a weight vector: w =( w u1 u 2, w u1 u 3,..., w u1 u n, w u2 u 3,..., w u2 u n,.... w un 1 u n, θ u1, θ u2,..., θ un ). Construct input vectors for a threshold logic unit z 2 =(act (p) u 1, 0,...,0, }{{} n 2zeros Apply Delta rule training until convergence. act (p) u 3,...,act (p) u n,...0, 1, 0,...,0 ). }{{} n 2zeros Rudolf Kruse Neural Networks 24
25 Demonstration Software: xhfn/whfn Demonstration of Hopfield networks as associative memory: Visualization of the association/recognition process Two-dimensional networks of arbitrary size Rudolf Kruse Neural Networks 25
26 Hopfield Networks: Solving Optimization Problems Use energy minimization to solve optimization problems General procedure: Transform function to optimize into a function to minimize. Transform function into the form of an energy function of a Hopfield network. Read the weights and threshold values from the energy function. Construct the corresponding Hopfield network. Initialize Hopfield network randomly and update until convergence. Read solution from the stable state reached. Repeat several times and use best solution found. Rudolf Kruse Neural Networks 26
27 Hopfield Networks: Activation Transformation AHopfieldnetworkmaybedefinedeitherwithactivations 1and1orwithactivations 0and1.Thenetworkscanbetransformedintoeachother. From act u { 1, 1} to act u {0, 1}: From act u {0, 1} to act u { 1, 1}: wuv 0 = 2wuv and θu 0 = θu + wuv v U {u} w uv = 1 2 w0 uv and θ u = θ 0 u 1 2 wuv. 0 v U {u} Rudolf Kruse Neural Networks 27
28 Hopfield Networks: Solving Optimization Problems Combination lemma: Let two Hopfield networks on the same set U of neurons with weights w uv, (i) thresholdvaluesθ u (i) and energy functions E i = 1 2 u U v U {u} w (i) uv act u act v + u U θ (i) u act u, i =1, 2, be given. Furthermore let a, b IR. Then E = ae 1 +be 2 is the energy function of the Hopfield network on the neurons in U that has the weights w uv = aw uv (1) + bw uv (2) and the threshold values θ u = aθ u (1) + bθ u (2). Proof: Just do the computations. Idea: Additional conditions can be formalized separately and incorporated later. Rudolf Kruse Neural Networks 28
29 Hopfield Networks: Solving Optimization Problems Example: Traveling salesman problem Idea: Represent tour by a matrix city step An element a ij of the matrix is 1 if the i-th city is visited in the j-th step and 0 otherwise. Each matrix element will be represented by a neuron. Rudolf Kruse Neural Networks 29
30 Hopfield Networks: Solving Optimization Problems Minimization of the tour length E 1 = n n n j 1 =1 j 2 =1 i=1 Double summation over steps (index i) needed: where E 1 = (i 1,j 1 ) {1,...,n} 2 d j1 j 2 m ij1 m (i mod n)+1,j2. (i 2,j 2 ) {1,...,n} 2 d j1 j 2 δ (i1 mod n)+1,i 2 m i1 j 1 m i2 j 2, δ ab = Symmetric version of the energy function: E 1 = 1 2 (i 1,j 1 ) {1,...,n} 2 (i 2,j 2 ) {1,...,n} 2 { 1, if a = b, 0, otherwise. d j1 j 2 (δ (i1 mod n)+1,i 2 + δ i1,(i 2 mod n)+1 ) m i 1 j 1 m i2 j 2. Rudolf Kruse Neural Networks 30
31 Hopfield Networks: Solving Optimization Problems Additional conditions that have to be satisfied: Each city is visited on exactly one step of the tour: j {1,...,n} : n i=1 m ij =1, i.e., each column of the matrix contains exactly one 1. On each step of the tour exactly one city is visited: i {1,...,n} : n j=1 m ij =1, i.e., each row of the matrix contains exactly one 1. These conditions are incorporated by finding additional functions to optimize. Rudolf Kruse Neural Networks 31
32 Hopfield Networks: Solving Optimization Problems Formalization of first condition as a minimization problem: E2 = = = 2 n n n m ij 2 m ij +1 j=1 i=1 i=1 n n n m i1 j m i2 j j=1 i 1 =1 i 2 =1 n n n n n j=1 i 1 =1 i 2 =1 m i1 jm i2 j 2 2 j=1 i=1 n i=1 m ij + n. m ij +1 Double summation over cities (index i) needed: E 2 = (i 1,j 1 ) {1,...,n} 2 δ j1 j 2 m i1 j 1 m i2 j 2 2 m ij. (i 2,j 2 ) {1,...,n} 2 (i,j) {1,...,n} 2 Rudolf Kruse Neural Networks 32
33 Hopfield Networks: Solving Optimization Problems Resulting energy function: E 2 = 1 2 (i 1,j 1 ) {1,...,n} 2 (i 2,j 2 ) {1,...,n} 2 2δ j1 j 2 m i1 j 1 m i2 j 2 + (i,j) {1,...,n} 2 2m ij Second additional condition is handled in a completely analogous way: E 3 = 1 2 (i 1,j 1 ) {1,...,n} 2 (i 2,j 2 ) {1,...,n} 2 2δ i1 i 2 m i1 j 1 m i2 j 2 + (i,j) {1,...,n} 2 2m ij. Combining the energy functions: E = ae 1 + be 2 + ce 3 where b a = c a > 2 max d (j 1,j 2 ) {1,...,n} 2 j 1 j 2. Rudolf Kruse Neural Networks 33
34 Hopfield Networks: Solving Optimization Problems From the resulting energy function we can read the weights w (i1,j 1 )(i 2,j 2 ) = ad j 1 j 2 (δ (i1 mod n)+1,i 2 + δ i1,(i 2 mod n)+1 ) 2bδ j1 j }{{} 2 2cδ i1 i }{{}}{{ 2 } from E 1 from E 2 from E 3 and the threshold values: θ (i,j) = 0a }{{} from E 1 2b }{{} from E 2 2c }{{} = 2(b + c). from E 3 Problem: Random initialization and update until convergence not always leads to a matrix that represents a tour, leave alone an optimal one. Rudolf Kruse Neural Networks 34
Hopfield Networks and Boltzmann Machines. Christian Borgelt Artificial Neural Networks and Deep Learning 296
Hopfield Networks and Boltzmann Machines Christian Borgelt Artificial Neural Networks and Deep Learning 296 Hopfield Networks A Hopfield network is a neural network with a graph G = (U,C) that satisfies
More informationIn biological terms, memory refers to the ability of neural systems to store activity patterns and later recall them when required.
In biological terms, memory refers to the ability of neural systems to store activity patterns and later recall them when required. In humans, association is known to be a prominent feature of memory.
More information3.3 Discrete Hopfield Net An iterative autoassociative net similar to the nets described in the previous sections has been developed by Hopfield
3.3 Discrete Hopfield Net An iterative autoassociative net similar to the nets described in the previous sections has been developed by Hopfield (1982, 1984). - The net is a fully interconnected neural
More informationHertz, Krogh, Palmer: Introduction to the Theory of Neural Computation. Addison-Wesley Publishing Company (1991). (v ji (1 x i ) + (1 v ji )x i )
Symmetric Networks Hertz, Krogh, Palmer: Introduction to the Theory of Neural Computation. Addison-Wesley Publishing Company (1991). How can we model an associative memory? Let M = {v 1,..., v m } be a
More informationHopfield Neural Network and Associative Memory. Typical Myelinated Vertebrate Motoneuron (Wikipedia) Topic 3 Polymers and Neurons Lecture 5
Hopfield Neural Network and Associative Memory Typical Myelinated Vertebrate Motoneuron (Wikipedia) PHY 411-506 Computational Physics 2 1 Wednesday, March 5 1906 Nobel Prize in Physiology or Medicine.
More informationArtificial Intelligence Hopfield Networks
Artificial Intelligence Hopfield Networks Andrea Torsello Network Topologies Single Layer Recurrent Network Bidirectional Symmetric Connection Binary / Continuous Units Associative Memory Optimization
More informationUsing a Hopfield Network: A Nuts and Bolts Approach
Using a Hopfield Network: A Nuts and Bolts Approach November 4, 2013 Gershon Wolfe, Ph.D. Hopfield Model as Applied to Classification Hopfield network Training the network Updating nodes Sequencing of
More informationComputational Intelligence Lecture 6: Associative Memory
Computational Intelligence Lecture 6: Associative Memory Farzaneh Abdollahi Department of Electrical Engineering Amirkabir University of Technology Fall 2011 Farzaneh Abdollahi Computational Intelligence
More informationA. The Hopfield Network. III. Recurrent Neural Networks. Typical Artificial Neuron. Typical Artificial Neuron. Hopfield Network.
Part 3A: Hopfield Network III. Recurrent Neural Networks A. The Hopfield Network 1 2 Typical Artificial Neuron Typical Artificial Neuron connection weights linear combination activation function inputs
More informationA. The Hopfield Network. III. Recurrent Neural Networks. Typical Artificial Neuron. Typical Artificial Neuron. Hopfield Network.
III. Recurrent Neural Networks A. The Hopfield Network 2/9/15 1 2/9/15 2 Typical Artificial Neuron Typical Artificial Neuron connection weights linear combination activation function inputs output net
More informationCHAPTER 3. Pattern Association. Neural Networks
CHAPTER 3 Pattern Association Neural Networks Pattern Association learning is the process of forming associations between related patterns. The patterns we associate together may be of the same type or
More informationHopfield Neural Network
Lecture 4 Hopfield Neural Network Hopfield Neural Network A Hopfield net is a form of recurrent artificial neural network invented by John Hopfield. Hopfield nets serve as content-addressable memory systems
More informationHopfield networks. Lluís A. Belanche Soft Computing Research Group
Lluís A. Belanche belanche@lsi.upc.edu Soft Computing Research Group Dept. de Llenguatges i Sistemes Informàtics (Software department) Universitat Politècnica de Catalunya 2010-2011 Introduction Content-addressable
More informationMemories Associated with Single Neurons and Proximity Matrices
Memories Associated with Single Neurons and Proximity Matrices Subhash Kak Oklahoma State University, Stillwater Abstract: This paper extends the treatment of single-neuron memories obtained by the use
More informationCOMP304 Introduction to Neural Networks based on slides by:
COMP34 Introduction to Neural Networks based on slides by: Christian Borgelt http://www.borgelt.net/ Christian Borgelt Introduction to Neural Networks Motivation: Why (Artificial) Neural Networks? (Neuro-)Biology
More informationNeural networks: Unsupervised learning
Neural networks: Unsupervised learning 1 Previously The supervised learning paradigm: given example inputs x and target outputs t learning the mapping between them the trained network is supposed to give
More informationIntroduction to Artificial Neural Networks
Facultés Universitaires Notre-Dame de la Paix 27 March 2007 Outline 1 Introduction 2 Fundamentals Biological neuron Artificial neuron Artificial Neural Network Outline 3 Single-layer ANN Perceptron Adaline
More informationNeural Networks. Hopfield Nets and Auto Associators Fall 2017
Neural Networks Hopfield Nets and Auto Associators Fall 2017 1 Story so far Neural networks for computation All feedforward structures But what about.. 2 Loopy network Θ z = ቊ +1 if z > 0 1 if z 0 y i
More informationIntroduction to Neural Networks
Introduction to Neural Networks What are (Artificial) Neural Networks? Models of the brain and nervous system Highly parallel Process information much more like the brain than a serial computer Learning
More informationUsing Variable Threshold to Increase Capacity in a Feedback Neural Network
Using Variable Threshold to Increase Capacity in a Feedback Neural Network Praveen Kuruvada Abstract: The article presents new results on the use of variable thresholds to increase the capacity of a feedback
More informationLearning and Memory in Neural Networks
Learning and Memory in Neural Networks Guy Billings, Neuroinformatics Doctoral Training Centre, The School of Informatics, The University of Edinburgh, UK. Neural networks consist of computational units
More informationWeek 4: Hopfield Network
Week 4: Hopfield Network Phong Le, Willem Zuidema November 20, 2013 Last week we studied multi-layer perceptron, a neural network in which information is only allowed to transmit in one direction (from
More informationArtificial Neural Networks Examination, March 2004
Artificial Neural Networks Examination, March 2004 Instructions There are SIXTY questions (worth up to 60 marks). The exam mark (maximum 60) will be added to the mark obtained in the laborations (maximum
More informationHow can ideas from quantum computing improve or speed up neuromorphic models of computation?
Neuromorphic Computation: Architectures, Models, Applications Associative Memory Models with Adiabatic Quantum Optimization Kathleen Hamilton, Alexander McCaskey, Jonathan Schrock, Neena Imam and Travis
More informationNeural Nets and Symbolic Reasoning Hopfield Networks
Neural Nets and Symbolic Reasoning Hopfield Networks Outline The idea of pattern completion The fast dynamics of Hopfield networks Learning with Hopfield networks Emerging properties of Hopfield networks
More informationNeural Networks. Solving of Optimization Problems 12/30/2015
/30/05 Neural Networks Solving of Optimization Problems Lecture 5 The ability for parallel computation yields to the possibility to work with big amount of data. It allows neural networks to solve complex
More informationLearning in State-Space Reinforcement Learning CIS 32
Learning in State-Space Reinforcement Learning CIS 32 Functionalia Syllabus Updated: MIDTERM and REVIEW moved up one day. MIDTERM: Everything through Evolutionary Agents. HW 2 Out - DUE Sunday before the
More informationSimple Neural Nets For Pattern Classification
CHAPTER 2 Simple Neural Nets For Pattern Classification Neural Networks General Discussion One of the simplest tasks that neural nets can be trained to perform is pattern classification. In pattern classification
More informationLecture 7 Artificial neural networks: Supervised learning
Lecture 7 Artificial neural networks: Supervised learning Introduction, or how the brain works The neuron as a simple computing element The perceptron Multilayer neural networks Accelerated learning in
More information2- AUTOASSOCIATIVE NET - The feedforward autoassociative net considered in this section is a special case of the heteroassociative net.
2- AUTOASSOCIATIVE NET - The feedforward autoassociative net considered in this section is a special case of the heteroassociative net. - For an autoassociative net, the training input and target output
More informationNeural Networks Lecture 6: Associative Memory II
Neural Networks Lecture 6: Associative Memory II H.A Talebi Farzaneh Abdollahi Department of Electrical Engineering Amirkabir University of Technology Winter 2011. A. Talebi, Farzaneh Abdollahi Neural
More informationNeural Network Essentials 1
Neural Network Essentials Draft: Associative Memory Chapter Anthony S. Maida University of Louisiana at Lafayette, USA June 6, 205 Copyright c 2000 2003 by Anthony S. Maida Contents 0. Associative memory.............................
More informationSPSS, University of Texas at Arlington. Topics in Machine Learning-EE 5359 Neural Networks
Topics in Machine Learning-EE 5359 Neural Networks 1 The Perceptron Output: A perceptron is a function that maps D-dimensional vectors to real numbers. For notational convenience, we add a zero-th dimension
More informationIntroduction to Neural Networks
Introduction to Neural Networks Christian Borgelt Intelligent Data Analysis and Graphical Models Research Unit European Center for Soft Computing c/ Gonzalo Gutiérrez Quirós s/n, 33600 Mieres, Spain christian.borgelt@softcomputing.es
More informationHebbian Learning II. Robert Jacobs Department of Brain & Cognitive Sciences University of Rochester. July 20, 2017
Hebbian Learning II Robert Jacobs Department of Brain & Cognitive Sciences University of Rochester July 20, 2017 Goals Teach about one-half of an undergraduate course on Linear Algebra Understand when
More informationNeural networks. Chapter 20. Chapter 20 1
Neural networks Chapter 20 Chapter 20 1 Outline Brains Neural networks Perceptrons Multilayer networks Applications of neural networks Chapter 20 2 Brains 10 11 neurons of > 20 types, 10 14 synapses, 1ms
More informationC4 Phenomenological Modeling - Regression & Neural Networks : Computational Modeling and Simulation Instructor: Linwei Wang
C4 Phenomenological Modeling - Regression & Neural Networks 4040-849-03: Computational Modeling and Simulation Instructor: Linwei Wang Recall.. The simple, multiple linear regression function ŷ(x) = a
More informationCOGS Q250 Fall Homework 7: Learning in Neural Networks Due: 9:00am, Friday 2nd November.
COGS Q250 Fall 2012 Homework 7: Learning in Neural Networks Due: 9:00am, Friday 2nd November. For the first two questions of the homework you will need to understand the learning algorithm using the delta
More informationCMSC 421: Neural Computation. Applications of Neural Networks
CMSC 42: Neural Computation definition synonyms neural networks artificial neural networks neural modeling connectionist models parallel distributed processing AI perspective Applications of Neural Networks
More informationLogic Learning in Hopfield Networks
Logic Learning in Hopfield Networks Saratha Sathasivam (Corresponding author) School of Mathematical Sciences, University of Science Malaysia, Penang, Malaysia E-mail: saratha@cs.usm.my Wan Ahmad Tajuddin
More informationNeural networks: Associative memory
Neural networs: Associative memory Prof. Sven Lončarić sven.loncaric@fer.hr http://www.fer.hr/ipg 1 Overview of topics l Introduction l Associative memories l Correlation matrix as an associative memory
More informationPlan. Perceptron Linear discriminant. Associative memories Hopfield networks Chaotic networks. Multilayer perceptron Backpropagation
Neural Networks Plan Perceptron Linear discriminant Associative memories Hopfield networks Chaotic networks Multilayer perceptron Backpropagation Perceptron Historically, the first neural net Inspired
More informationIterative Autoassociative Net: Bidirectional Associative Memory
POLYTECHNIC UNIVERSITY Department of Computer and Information Science Iterative Autoassociative Net: Bidirectional Associative Memory K. Ming Leung Abstract: Iterative associative neural networks are introduced.
More informationNeural Networks. CSE 6363 Machine Learning Vassilis Athitsos Computer Science and Engineering Department University of Texas at Arlington
Neural Networks CSE 6363 Machine Learning Vassilis Athitsos Computer Science and Engineering Department University of Texas at Arlington 1 Perceptrons x 0 = 1 x 1 x 2 z = h w T x Output: z x D A perceptron
More informationCE213 Artificial Intelligence Lecture 14
CE213 Artificial Intelligence Lecture 14 Neural Networks: Part 2 Learning Rules -Hebb Rule - Perceptron Rule -Delta Rule Neural Networks Using Linear Units [ Difficulty warning: equations! ] 1 Learning
More information7.1 Basis for Boltzmann machine. 7. Boltzmann machines
7. Boltzmann machines this section we will become acquainted with classical Boltzmann machines which can be seen obsolete being rarely applied in neurocomputing. It is interesting, after all, because is
More informationPattern Association or Associative Networks. Jugal Kalita University of Colorado at Colorado Springs
Pattern Association or Associative Networks Jugal Kalita University of Colorado at Colorado Springs To an extent, learning is forming associations. Human memory associates similar items, contrary/opposite
More informationOn the Hopfield algorithm. Foundations and examples
General Mathematics Vol. 13, No. 2 (2005), 35 50 On the Hopfield algorithm. Foundations and examples Nicolae Popoviciu and Mioara Boncuţ Dedicated to Professor Dumitru Acu on his 60th birthday Abstract
More informationSample Exam COMP 9444 NEURAL NETWORKS Solutions
FAMILY NAME OTHER NAMES STUDENT ID SIGNATURE Sample Exam COMP 9444 NEURAL NETWORKS Solutions (1) TIME ALLOWED 3 HOURS (2) TOTAL NUMBER OF QUESTIONS 12 (3) STUDENTS SHOULD ANSWER ALL QUESTIONS (4) QUESTIONS
More informationChristian Mohr
Christian Mohr 20.12.2011 Recurrent Networks Networks in which units may have connections to units in the same or preceding layers Also connections to the unit itself possible Already covered: Hopfield
More informationArtificial Neural Networks Examination, June 2005
Artificial Neural Networks Examination, June 2005 Instructions There are SIXTY questions. (The pass mark is 30 out of 60). For each question, please select a maximum of ONE of the given answers (either
More informationAssociative Memories (I) Hopfield Networks
Associative Memories (I) Davide Bacciu Dipartimento di Informatica Università di Pisa bacciu@di.unipi.it Applied Brain Science - Computational Neuroscience (CNS) A Pun Associative Memories Introduction
More informationMachine Learning. Neural Networks
Machine Learning Neural Networks Bryan Pardo, Northwestern University, Machine Learning EECS 349 Fall 2007 Biological Analogy Bryan Pardo, Northwestern University, Machine Learning EECS 349 Fall 2007 THE
More informationCHALMERS, GÖTEBORGS UNIVERSITET. EXAM for ARTIFICIAL NEURAL NETWORKS. COURSE CODES: FFR 135, FIM 720 GU, PhD
CHALMERS, GÖTEBORGS UNIVERSITET EXAM for ARTIFICIAL NEURAL NETWORKS COURSE CODES: FFR 135, FIM 72 GU, PhD Time: Place: Teachers: Allowed material: Not allowed: October 23, 217, at 8 3 12 3 Lindholmen-salar
More informationNeural Nets in PR. Pattern Recognition XII. Michal Haindl. Outline. Neural Nets in PR 2
Neural Nets in PR NM P F Outline Motivation: Pattern Recognition XII human brain study complex cognitive tasks Michal Haindl Faculty of Information Technology, KTI Czech Technical University in Prague
More informationIn this section, we review some basics of modeling via linear algebra: finding a line of best fit, Hebbian learning, pattern classification.
Linear Models In this section, we review some basics of modeling via linear algebra: finding a line of best fit, Hebbian learning, pattern classification. Best Fitting Line In this section, we examine
More informationMultilayer Perceptron
Outline Hong Chang Institute of Computing Technology, Chinese Academy of Sciences Machine Learning Methods (Fall 2012) Outline Outline I 1 Introduction 2 Single Perceptron 3 Boolean Function Learning 4
More informationStorage Capacity of Letter Recognition in Hopfield Networks
Storage Capacity of Letter Recognition in Hopfield Networks Gang Wei (gwei@cs.dal.ca) Zheyuan Yu (zyu@cs.dal.ca) Faculty of Computer Science, Dalhousie University, Halifax, N.S., Canada B3H 1W5 Abstract:
More informationCOMP 551 Applied Machine Learning Lecture 14: Neural Networks
COMP 551 Applied Machine Learning Lecture 14: Neural Networks Instructor: Ryan Lowe (ryan.lowe@mail.mcgill.ca) Slides mostly by: Class web page: www.cs.mcgill.ca/~hvanho2/comp551 Unless otherwise noted,
More informationArtificial Neural Networks and Deep Learning
Artificial Neural Networks and Deep Learning Christian Borgelt School of Computer Science University of Konstanz Universitätsstraße 0, 78457 Konstanz, Germany christian.borgelt@uni-konstanz.de christian@borgelt.net
More informationHopfield Networks. (Excerpt from a Basic Course at IK 2008) Herbert Jaeger. Jacobs University Bremen
Hopfield Networks (Excerpt from a Basic Course at IK 2008) Herbert Jaeger Jacobs University Bremen Building a model of associative memory should be simple enough... Our brain is a neural network Individual
More informationArtificial Neural Network
Artificial Neural Network Contents 2 What is ANN? Biological Neuron Structure of Neuron Types of Neuron Models of Neuron Analogy with human NN Perceptron OCR Multilayer Neural Network Back propagation
More informationNetworks of McCulloch-Pitts Neurons
s Lecture 4 Netorks of McCulloch-Pitts Neurons The McCulloch and Pitts (M_P) Neuron x x sgn x n Netorks of M-P Neurons One neuron can t do much on its on, but a net of these neurons x i x i i sgn i ij
More informationOptimization of Quadratic Forms: NP Hard Problems : Neural Networks
1 Optimization of Quadratic Forms: NP Hard Problems : Neural Networks Garimella Rama Murthy, Associate Professor, International Institute of Information Technology, Gachibowli, HYDERABAD, AP, INDIA ABSTRACT
More informationCE213 Artificial Intelligence Lecture 13
CE213 Artificial Intelligence Lecture 13 Neural Networks What is a Neural Network? Why Neural Networks? (New Models and Algorithms for Problem Solving) McCulloch-Pitts Neural Nets Learning Using The Delta
More informationNeural networks. Chapter 19, Sections 1 5 1
Neural networks Chapter 19, Sections 1 5 Chapter 19, Sections 1 5 1 Outline Brains Neural networks Perceptrons Multilayer perceptrons Applications of neural networks Chapter 19, Sections 1 5 2 Brains 10
More informationAn artificial neural networks (ANNs) model is a functional abstraction of the
CHAPER 3 3. Introduction An artificial neural networs (ANNs) model is a functional abstraction of the biological neural structures of the central nervous system. hey are composed of many simple and highly
More informationArtificial Neural Network and Fuzzy Logic
Artificial Neural Network and Fuzzy Logic 1 Syllabus 2 Syllabus 3 Books 1. Artificial Neural Networks by B. Yagnanarayan, PHI - (Cover Topologies part of unit 1 and All part of Unit 2) 2. Neural Networks
More informationSynaptic Plasticity. Introduction. Biophysics of Synaptic Plasticity. Functional Modes of Synaptic Plasticity. Activity-dependent synaptic plasticity:
Synaptic Plasticity Introduction Dayan and Abbott (2001) Chapter 8 Instructor: Yoonsuck Choe; CPSC 644 Cortical Networks Activity-dependent synaptic plasticity: underlies learning and memory, and plays
More informationComputational Intelligence Winter Term 2017/18
Computational Intelligence Winter Term 207/8 Prof. Dr. Günter Rudolph Lehrstuhl für Algorithm Engineering (LS ) Fakultät für Informatik TU Dortmund Plan for Today Single-Layer Perceptron Accelerated Learning
More informationContent-Addressable Memory Associative Memory Lernmatrix Association Heteroassociation Learning Retrieval Reliability of the answer
Associative Memory Content-Addressable Memory Associative Memory Lernmatrix Association Heteroassociation Learning Retrieval Reliability of the answer Storage Analysis Sparse Coding Implementation on a
More information6. APPLICATION TO THE TRAVELING SALESMAN PROBLEM
6. Application to the Traveling Salesman Problem 92 6. APPLICATION TO THE TRAVELING SALESMAN PROBLEM The properties that have the most significant influence on the maps constructed by Kohonen s algorithm
More information( ) T. Reading. Lecture 22. Definition of Covariance. Imprinting Multiple Patterns. Characteristics of Hopfield Memory
Part 3: Autonomous Agents /8/07 Reading Lecture 22 Flake, ch. 20 ( Genetics and Evolution ) /8/07 /8/07 2 Imprinting Multiple Patterns Let x, x 2,, x p be patterns to be imprinted Define the sum-of-outer-products
More informationIterative face image feature extraction with Generalized Hebbian Algorithm and a Sanger-like BCM rule
Iterative face image feature extraction with Generalized Hebbian Algorithm and a Sanger-like BCM rule Clayton Aldern (Clayton_Aldern@brown.edu) Tyler Benster (Tyler_Benster@brown.edu) Carl Olsson (Carl_Olsson@brown.edu)
More informationHopfield and Potts-Hopfield Networks
Hopfield and Potts-Hopfield Networks Andy Somogyi Dec 10, 2010 1 Introduction Many early models of neural networks lacked mathematical rigor or analysis. They were, by and large, ad hoc models. John Hopfield
More informationArtificial Neural Networks Examination, March 2002
Artificial Neural Networks Examination, March 2002 Instructions There are SIXTY questions (worth up to 60 marks). The exam mark (maximum 60) will be added to the mark obtained in the laborations (maximum
More informationMaster Recherche IAC TC2: Apprentissage Statistique & Optimisation
Master Recherche IAC TC2: Apprentissage Statistique & Optimisation Alexandre Allauzen Anne Auger Michèle Sebag LIMSI LRI Oct. 4th, 2012 This course Bio-inspired algorithms Classical Neural Nets History
More information7 Recurrent Networks of Threshold (Binary) Neurons: Basis for Associative Memory
Physics 178/278 - David Kleinfeld - Winter 2019 7 Recurrent etworks of Threshold (Binary) eurons: Basis for Associative Memory 7.1 The network The basic challenge in associative networks, also referred
More informationComputational Intelligence
Plan for Today Single-Layer Perceptron Computational Intelligence Winter Term 00/ Prof. Dr. Günter Rudolph Lehrstuhl für Algorithm Engineering (LS ) Fakultät für Informatik TU Dortmund Accelerated Learning
More informationCSC321 Lecture 7: Optimization
CSC321 Lecture 7: Optimization Roger Grosse Roger Grosse CSC321 Lecture 7: Optimization 1 / 25 Overview We ve talked a lot about how to compute gradients. What do we actually do with them? Today s lecture:
More informationCOMP9444 Neural Networks and Deep Learning 11. Boltzmann Machines. COMP9444 c Alan Blair, 2017
COMP9444 Neural Networks and Deep Learning 11. Boltzmann Machines COMP9444 17s2 Boltzmann Machines 1 Outline Content Addressable Memory Hopfield Network Generative Models Boltzmann Machine Restricted Boltzmann
More informationLast update: October 26, Neural networks. CMSC 421: Section Dana Nau
Last update: October 26, 207 Neural networks CMSC 42: Section 8.7 Dana Nau Outline Applications of neural networks Brains Neural network units Perceptrons Multilayer perceptrons 2 Example Applications
More informationCatastrophic forgetting in simple networks: an analysis of the pseudorehearsal solution
Network: Comput. Neural Syst. 0 (999) 227 236. Printed in the UK PII: S0954-898X(99)9972-X Catastrophic forgetting in simple networks: an analysis of the pseudorehearsal solution Marcus Frean and Anthony
More informationNeural Networks Lecture 2:Single Layer Classifiers
Neural Networks Lecture 2:Single Layer Classifiers H.A Talebi Farzaneh Abdollahi Department of Electrical Engineering Amirkabir University of Technology Winter 2011. A. Talebi, Farzaneh Abdollahi Neural
More informationCSC321 Lecture 8: Optimization
CSC321 Lecture 8: Optimization Roger Grosse Roger Grosse CSC321 Lecture 8: Optimization 1 / 26 Overview We ve talked a lot about how to compute gradients. What do we actually do with them? Today s lecture:
More informationNeural Networks for Machine Learning. Lecture 2a An overview of the main types of neural network architecture
Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network architecture Geoffrey Hinton with Nitish Srivastava Kevin Swersky Feed-forward neural networks These are
More informationAdministration. Registration Hw3 is out. Lecture Captioning (Extra-Credit) Scribing lectures. Questions. Due on Thursday 10/6
Administration Registration Hw3 is out Due on Thursday 10/6 Questions Lecture Captioning (Extra-Credit) Look at Piazza for details Scribing lectures With pay; come talk to me/send email. 1 Projects Projects
More information7 Rate-Based Recurrent Networks of Threshold Neurons: Basis for Associative Memory
Physics 178/278 - David Kleinfeld - Fall 2005; Revised for Winter 2017 7 Rate-Based Recurrent etworks of Threshold eurons: Basis for Associative Memory 7.1 A recurrent network with threshold elements The
More informationNeural Networks DWML, /25
DWML, 2007 /25 Neural networks: Biological and artificial Consider humans: Neuron switching time 0.00 second Number of neurons 0 0 Connections per neuron 0 4-0 5 Scene recognition time 0. sec 00 inference
More informationData Mining Part 5. Prediction
Data Mining Part 5. Prediction 5.5. Spring 2010 Instructor: Dr. Masoud Yaghini Outline How the Brain Works Artificial Neural Networks Simple Computing Elements Feed-Forward Networks Perceptrons (Single-layer,
More informationCSE 5526: Introduction to Neural Networks Hopfield Network for Associative Memory
CSE 5526: Introduction to Neural Networks Hopfield Network for Associative Memory Part VII 1 The basic task Store a set of fundamental memories {ξξ 1, ξξ 2,, ξξ MM } so that, when presented a new pattern
More informationLecture 4: Feed Forward Neural Networks
Lecture 4: Feed Forward Neural Networks Dr. Roman V Belavkin Middlesex University BIS4435 Biological neurons and the brain A Model of A Single Neuron Neurons as data-driven models Neural Networks Training
More informationQ520: Answers to the Homework on Hopfield Networks. 1. For each of the following, answer true or false with an explanation:
Q50: Answers to the Homework on Hopfield Networks 1. For each of the following, answer true or false with an explanation: a. Fix a Hopfield net. If o and o are neighboring observation patterns then Φ(
More informationarxiv: v2 [nlin.ao] 19 May 2015
Efficient and optimal binary Hopfield associative memory storage using minimum probability flow arxiv:1204.2916v2 [nlin.ao] 19 May 2015 Christopher Hillar Redwood Center for Theoretical Neuroscience University
More information18.6 Regression and Classification with Linear Models
18.6 Regression and Classification with Linear Models 352 The hypothesis space of linear functions of continuous-valued inputs has been used for hundreds of years A univariate linear function (a straight
More informationPerceptron. (c) Marcin Sydow. Summary. Perceptron
Topics covered by this lecture: Neuron and its properties Mathematical model of neuron: as a classier ' Learning Rule (Delta Rule) Neuron Human neural system has been a natural source of inspiration for
More informationLecture 5: Recurrent Neural Networks
1/25 Lecture 5: Recurrent Neural Networks Nima Mohajerin University of Waterloo WAVE Lab nima.mohajerin@uwaterloo.ca July 4, 2017 2/25 Overview 1 Recap 2 RNN Architectures for Learning Long Term Dependencies
More informationMarkov Chains, Random Walks on Graphs, and the Laplacian
Markov Chains, Random Walks on Graphs, and the Laplacian CMPSCI 791BB: Advanced ML Sridhar Mahadevan Random Walks! There is significant interest in the problem of random walks! Markov chain analysis! Computer
More informationNeural Networks: Basics. Darrell Whitley Colorado State University
Neural Networks: Basics Darrell Whitley Colorado State University In the Beginning: The Perceptron X1 W W 1,1 1,2 X2 W W 2,1 2,2 W source, destination In the Beginning: The Perceptron The Perceptron Learning
More informationArtifical Neural Networks
Neural Networks Artifical Neural Networks Neural Networks Biological Neural Networks.................................. Artificial Neural Networks................................... 3 ANN Structure...........................................
More information