Hopfield Networks. (Excerpt from a Basic Course at IK 2008) Herbert Jaeger. Jacobs University Bremen
|
|
- Henry King
- 6 years ago
- Views:
Transcription
1 Hopfield Networks (Excerpt from a Basic Course at IK 2008) Herbert Jaeger Jacobs University Bremen
2 Building a model of associative memory should be simple enough... Our brain is a neural network Individual neurons are quite well understood Almost universally shared belief: memories are coded in synaptic connectivity So it only remains to find out how the memories go there and how they are retrieved
3 Our topic for today How can "information" be "coded" in the "connectivity" of "neural networks"? Eerh... hrmm... What does this mean? All of these concepts are so imprecise... Somebody's gotta set the rules. We need a decision. A tough decision. And stick to it.
4 The Sheriff says... We can store discrete items in memory (finitely many, individual, fundamental patterns) Fundamental patterns are addressed by auto-association: the fundamental pattern is reconstituted from a "similar" cue input Example: "similar" = "corrupted": Pattern restauration Example: "similar" = "partial": Pattern completion Images from: Hertz et al 99
5 Other "similarity" relationships cue - pattern Thinking only of visual patterns for simplicity... "similar" = distorted "similar" = shifted, mirrored, rotated "similar" = B/W (pattern in color) "similar" = line drawing (pattern photo) "similar" = preceding in time (pattern appears in animated sequence) "similar" = still (cueing an animated scene)
6 The rolling-downhill-ball model of memory Consider space of all possible neural "pattern states" (in fig.: 2-dim vector space spanned by V, V2) Within this space, some points are the fundamental patterns ( i in fig.) Above the pattern space, an energy landscape (or a potential) E is defined. Fundamental patterns lie at the minima of the energy. Any pattern may serve as cue. The process of associative retrieval of a pattern from a "similar" cue is determined by gradient descent from E( ) to E( ). E 2 3
7 Aerial view of the same This contour plot shows the energy landscape like in a map. The pattern space becomes divided into basins of attraction. E.g., all cue patterns that are "attracted" by 2 form the basin of attraction of the fixed point attractor One may say that all these are instances of the category (concept, class) represented by 2. 3
8 Same, with "real" patterns Pixel images from Haykin 999
9 Structure of a Hopfield network (HN) A HN is made of N binary neurons (in figure: N = 4) Each neuron is connected to all other neurons (except itself - no auto-feedback) Connection between neurons i and j is symmetric (same as connection between j and i) and has a weight w ij = w ji R. At a given time, neuron i has state s i {, }. The entire network has a state S = (s,..., s N ) T (written as a column vector) A simple demo HN w 3 = w 3 = 0.5 s = w 2 = w 2 = 3 s 3 = w 4 = w 4 = 0.2 w 32 = w 23 = s 2 = s 4 = w 34 = w 43 = 2 s s2 S = = s 3 s4 w 42 = w 24 = 0.
10 Energy of a state By definition, a state S = (s,..., s N ) T has an energy s = 3 s 2 = E( S) = N w ij i, j i< j =,..., N s i s j Example right: E(S) = /4 ( w 2 w 3 w 4 + w 23 + w 24 + w 34 ) = /4 ( ) = 0.35 Electric metaphor: higher energy means more "neighboring opposite charges" (almost like a charged battery) s 3 = 2 s s2 S = = s 3 s4 s 4 =
11 A closeup on the neuron model The (discrete-time) HN is made of McCulloch-Pitts neurons (same as the perceptron) Update rule: Neuron i first sums its connection-weighted inputs, then takes difference to a threshold i, then passes the sum through a binary decision function, the sign function sgn(x) = if x < 0, sgn(x) = + if x 0 Biology view Engineering view s ( t i + ) = sgn w j=... N j i ij s j ( t) Θ i Math view (Figures from the Maida lecture notes)
12 Convention: in HNs, all i = 0 Updating neurons and networks Single neuron update, for instance of neuron 3: s 3 ( t + ) = sgn w j=,2,4 ( t) = sgn( 0.5*+ * 2*) = sgn(.5) = ij s j s = s 2 = 0. Entire network update:. Pick one neuron at random (stochastic choice) 2. Update it as above (deterministic update) 3. Iterate HNs evolve over time by individual updates of randomly picked neurons. s 3 = 2 s s2 S = = s 3 s4 s 4 =
13 Updating neurons and networks 2 From previous slide: update of s 3 was s 3 (t) = s 3 (t+) = This is a network update S( t) = S( t + ) = s = s 2 = 0. Energy at time t was E(S(t)) = Calculate: E(S(t+)) =.. Observation: E(S(t+)) < E(S(t)). s 3 = 2 s 4 = Fact: whenever S(t+) S(t), then E(S(t+)) < E(S(t)). "All state changes reduce the energy in a Hopfield network." s s2 S = = s 3 s4
14 Representing patterns by HN states A "pattern" is identified with a state (thus, every neuron takes part in representing a pattern) HNs "patterns" are therefore just N-dimensional {-,} vectors. To represent real-world patterns, one must code them as binary vectors. Example: The image could be coded as = S
15 HN as memory: Problem statement Given: p fundamental patterns,..., p, coded as N-dimensional {-,}- vectors. Wanted: a HN with N neurons, which has these fundamental patterns as the minima of its energy landscape. E(S) S Reward: we would have a neural network, which when started in some pattern (state) "similar" to a fundamental pattern, would evolve toward it by the stochastic state update and the energy minimization principle.
16 Remember our intuition
17 Hypercube state space of HNs The states of a HN are not continuous but discrete. A HN with N neurons can host 2 N different states - a finite number. These states can conveniently be arranged at the corners of an N-dimensional hypercube. N = 2 N = ) ( ) ( N = 3 N = 4
18 "Rolling downhill" in a hypercube Just to give you an impression... Pattern states in a N = 6 HN (slightly misleading though, images have N = 20!) Indicates energy at state (hypercube corner) Single-neuron update (flips one component of state vector)
19 Problem statement, again Given: p fundamental patterns,..., p, coded as N-dimensional {-,}-vectors. Wanted: a HN with N neurons, which has these fundamental patterns as the minima of the energy values at hypercube corners.... Seems a tough problem.
20 Solution: the HN should learn that by itself! Observation: the dynamic behaviour of a HN depends only on the weights w ij. Idea: train the weights by "showing" the fundamental patterns,..., p repeatedly to the network. At each exposure, the weights are adapted a little bit such that the energy for that pattern is a bit reduced. After numerous exposures, the energy landscape should have deep troughs at the states,..., p. Where do we get such a clever weight adaptation rule from?
21 Donald O. Hebb ( ) Started out as English teacher Turned to an academic carreer in psychology 949 The Organization of Behavior: A Neuropsychological Theory williamcalvin.com/bk9/bk9inter.htm
22 Hebb's postulate (949) "When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A s efficiency, as one of the cells firing B, is increased." (Hebb, The Organization of Behavior, 949) Also known as, "Cells that fire together wire together" Or simply as, "Hebb's learning rule"
23 Training HNs with Hebb's rule Hebb's rule spells out like this: "Presenting = (,..., N ) to the network" means just this, "set the N neuron states equal to (,..., N )." The neurobiological, Hebbian version of adapation: " i and j fire together if sgn( i ) = sgn( j ). Then, increase w ij a bit." Corresponds to energy reduction. Remember, Thus, if sgn( i ) = sgn( j ), increasing w ij reduces E( ). µ µ E( ξµ ) = w ij ξi ξ j. N i, j,..., i< j = N This leads to the following training scheme.. Present,..., p to the HN repeatedly (in random or cyclic order). 2. When is presented, update all weights by w ij w ij + λ i j (where λ is a small learning rate)
24 Training HNs with Hebb's rule, continued Training scheme (repeated):. Present,..., p to the HN repeatedly. 2. When is presented, update all weights by w ij w ij + λ i j Effect: On each pattern presentation of some, the energy of the corresponding state is lowered a bit. HOPE: We will eventually get a network that has energy troughs at the fundamental pattern states (and thus can serve as a "ball-rolling-downhill" memory)
25 A shortcut to the iterative training scheme Fact: the iterative learning scheme will converge (up to an irrelevant scaling constant) to a unique set of weights. Luck: there exists a simple one-shot computation for this asymptotic weight set, given by w ij p µ = ξ µ i ξ j N µ=
26 Summary so far Given p fundamental patterns,..., p (coded as N-dimensional {-, } vectors), we can "store" them into an N-dimensional HN by setting the weights according to w ij p µ = ξ µ i ξ j N µ= These weights are equivalent (up to irrelevant scaling) to the weights that we would ultimately get by using Hebbian learning, which at each step "tries" to reduce the energy of one of the states.
27 Natural questions Do we really get local energy minima at the fundamental pattern states? How many patterns can we store? Do we get energy minima only at the fundamental pattern states (or do we also create other local minima, that is, "false" memories)? Answers to these questions are known, and the analysis behind all of this has made Hopfield Networks so famous.
28 Q: fundamental patterns = local minima? Test for this: present a fundamental pattern to the trained network. If (and only if) the state corresponds to a local minimum, then all bits i of that state vector must be stable under the state update rule, that is ξ µ i µ µ ( t + ) = sgn w ξ ( t) = ξ ( t) Outcome: unfortunately, not always are all fundamental patterns stable. Analytic result: If p patterns are stored in a size N network, the probability that a randomly picked bit in a randomly picked pattern will flip when updated, is j i ij j i P(i unstable) = Φ( N p ) Note: Φ(a) is the shaded area under the standard Gaussian distribution a 0
29 More on stability... Situation: p patterns are stored in a size N network. Then the probability that a pattern bit is unstable when updated is P(i unstable) = Φ( N p ) Φ(a) a 0 Consequence: the pattern bit stability is related to p/n, called the load of the HN. The more patterns stored relative to the network size, the higher the chances that bits of fundamental patterns may flip. Beware of avalanches! The bit stability refers only to one isolated update within a presented pattern. But... If the entire network ist "run" by iterated state bit updates, flipped bits may pull further bits with them...
30 Avalanche stability Situation: p patterns are stored in a size N network. Pick a pattern and present it to the network (that is, make it the starting state). Some of its bits may be unstable under bit update. Two things may happen under iterated stochastic network update:. While some bits may flip, their flips don't induce more and more flips - the dynamics stabilizes in a pattern not too different from the original one. 2. The unstable bits trigger an avalanche, just like in a nuclear chain reaction. Analysis: whether or not avalanches occur depends on the load p/n. In the limit of large N, at a load of p/n > 0.38 every pattern becomes avalanche-instable. Figure: % of changed pattern bits under iterated network update, vs. load (here denoted by ). (D. J. Amit et al., Storing infinite numbers of patterns in a spin-glass model of neural networks. Phys. Rev. Let. 55(4), 985)
31 Avalanche stability: comments Load below 0.38: essentially stable memories Load above 0.38: suddenly all memories are completely unstable p/n = 0.38 is a critical value of the load parameter; when it is crossed, the behaviour of the system changes abruptly and dramatically. This sudden change of qualitative behaviour at critical values is common in nature; here we have a case of a phase transition.
32 Q: are there "false memories" (other local minima)? Let's train a 20-neuron HN on these 8 fundamental patterns: Which minimal-energy patterns will emerge in the HN? (how many local troughs will the energy landscape have?)
33 Some "false memories" Starting from 43,970 random states ended in these final stable states (from Haykin 998). There are three kinds of "false memories":. Inverted states (occur necessarily) 2. Mixtures of an odd number of fundamental patterns (example: mixture of, 4, 9 patterns) 3. spurious states (aka spin glass states) which are uncorrelated to stored patterns
34 False memories and network load For all loads p/n: stable spin glass states exist. For p/n > 0.38: spin glass states are the only stable ones. For 0 < p/n < 0.38: stable states close to desired fundamental patterns exist. For 0 < p/n < 0.05: pattern-related stable states have lower energy than spin glass states. For 0.05 < p/n < 0.38: spin glass states dominate (some of them have lower energy than pattern-related states) For 0 < p/n < 0.03: additional mixture states exist, with energies not quite as low as the pattern-related states. In sum, a HN works really well only for a load 0.03 < p/n < Results on this slide reported after: MacKay 2003 (chapter 42).
35 Hopfield networks: Pro's Simple model, can be mathematically analysed Biologically not immediately unrealistic Has strongly influenced how neuroscientists think about memory Connections to other field of computational physics (spin glass models) Are robust against "brain damage" (not discussed here) Have historically helped to salvage neural network research, and J. J. Hopfield was traded over several years as a Nobel prize candidate
36 Hopfield networks: Con's Small memory capacity: can store only about 5-0% of Nr. of patterns compared to network size (but is this really "small" -? Considering how many neurons we have... -?) All the "nice" results hold for uncorrelated fundamental patterns - an unrealistic assumption Not technically useful Unwanted spurious, inverted and superimposed memories Has strongly influenced how neuroscientists think about memory
37 Variants, ramifications Continuous-time, continuous-state HNs (see Hopfield chapters in Haykin's and MacKay's textbooks) Have also been used to tackle hard optimization problems (e.g., google "Traveling Salesman" + "Hopfield network") Still and repeatedly and always anew the subject of studies. Biologically plausible connectivity patterns have been studied, as well as small-world connectivity patterns (Davey et al 2005)
A. The Hopfield Network. III. Recurrent Neural Networks. Typical Artificial Neuron. Typical Artificial Neuron. Hopfield Network.
III. Recurrent Neural Networks A. The Hopfield Network 2/9/15 1 2/9/15 2 Typical Artificial Neuron Typical Artificial Neuron connection weights linear combination activation function inputs output net
More informationA. The Hopfield Network. III. Recurrent Neural Networks. Typical Artificial Neuron. Typical Artificial Neuron. Hopfield Network.
Part 3A: Hopfield Network III. Recurrent Neural Networks A. The Hopfield Network 1 2 Typical Artificial Neuron Typical Artificial Neuron connection weights linear combination activation function inputs
More informationHopfield Neural Network and Associative Memory. Typical Myelinated Vertebrate Motoneuron (Wikipedia) Topic 3 Polymers and Neurons Lecture 5
Hopfield Neural Network and Associative Memory Typical Myelinated Vertebrate Motoneuron (Wikipedia) PHY 411-506 Computational Physics 2 1 Wednesday, March 5 1906 Nobel Prize in Physiology or Medicine.
More information7 Rate-Based Recurrent Networks of Threshold Neurons: Basis for Associative Memory
Physics 178/278 - David Kleinfeld - Fall 2005; Revised for Winter 2017 7 Rate-Based Recurrent etworks of Threshold eurons: Basis for Associative Memory 7.1 A recurrent network with threshold elements The
More information7 Recurrent Networks of Threshold (Binary) Neurons: Basis for Associative Memory
Physics 178/278 - David Kleinfeld - Winter 2019 7 Recurrent etworks of Threshold (Binary) eurons: Basis for Associative Memory 7.1 The network The basic challenge in associative networks, also referred
More informationIn biological terms, memory refers to the ability of neural systems to store activity patterns and later recall them when required.
In biological terms, memory refers to the ability of neural systems to store activity patterns and later recall them when required. In humans, association is known to be a prominent feature of memory.
More information( ) T. Reading. Lecture 22. Definition of Covariance. Imprinting Multiple Patterns. Characteristics of Hopfield Memory
Part 3: Autonomous Agents /8/07 Reading Lecture 22 Flake, ch. 20 ( Genetics and Evolution ) /8/07 /8/07 2 Imprinting Multiple Patterns Let x, x 2,, x p be patterns to be imprinted Define the sum-of-outer-products
More informationUsing a Hopfield Network: A Nuts and Bolts Approach
Using a Hopfield Network: A Nuts and Bolts Approach November 4, 2013 Gershon Wolfe, Ph.D. Hopfield Model as Applied to Classification Hopfield network Training the network Updating nodes Sequencing of
More informationArtificial Intelligence Hopfield Networks
Artificial Intelligence Hopfield Networks Andrea Torsello Network Topologies Single Layer Recurrent Network Bidirectional Symmetric Connection Binary / Continuous Units Associative Memory Optimization
More informationNeural Nets and Symbolic Reasoning Hopfield Networks
Neural Nets and Symbolic Reasoning Hopfield Networks Outline The idea of pattern completion The fast dynamics of Hopfield networks Learning with Hopfield networks Emerging properties of Hopfield networks
More informationNeural Networks. Hopfield Nets and Auto Associators Fall 2017
Neural Networks Hopfield Nets and Auto Associators Fall 2017 1 Story so far Neural networks for computation All feedforward structures But what about.. 2 Loopy network Θ z = ቊ +1 if z > 0 1 if z 0 y i
More informationNeural Networks for Machine Learning. Lecture 11a Hopfield Nets
Neural Networks for Machine Learning Lecture 11a Hopfield Nets Geoffrey Hinton Nitish Srivastava, Kevin Swersky Tijmen Tieleman Abdel-rahman Mohamed Hopfield Nets A Hopfield net is composed of binary threshold
More informationCOMP9444 Neural Networks and Deep Learning 11. Boltzmann Machines. COMP9444 c Alan Blair, 2017
COMP9444 Neural Networks and Deep Learning 11. Boltzmann Machines COMP9444 17s2 Boltzmann Machines 1 Outline Content Addressable Memory Hopfield Network Generative Models Boltzmann Machine Restricted Boltzmann
More informationStorage Capacity of Letter Recognition in Hopfield Networks
Storage Capacity of Letter Recognition in Hopfield Networks Gang Wei (gwei@cs.dal.ca) Zheyuan Yu (zyu@cs.dal.ca) Faculty of Computer Science, Dalhousie University, Halifax, N.S., Canada B3H 1W5 Abstract:
More informationHopfield Network for Associative Memory
CSE 5526: Introduction to Neural Networks Hopfield Network for Associative Memory 1 The next few units cover unsupervised models Goal: learn the distribution of a set of observations Some observations
More informationHopfield Neural Network
Lecture 4 Hopfield Neural Network Hopfield Neural Network A Hopfield net is a form of recurrent artificial neural network invented by John Hopfield. Hopfield nets serve as content-addressable memory systems
More informationNeural Network Essentials 1
Neural Network Essentials Draft: Associative Memory Chapter Anthony S. Maida University of Louisiana at Lafayette, USA June 6, 205 Copyright c 2000 2003 by Anthony S. Maida Contents 0. Associative memory.............................
More informationChapter 10 Associative Memory Networks We are in for a surprise though!
We are in for a surprise though! 11 If we return to our calculated W based on the stored ξ and use x = (-1,1,-1,-1,-1,1) T as the input then we get the output y = (-1,1,-1,-1,-1,1) T, i.e. the same output
More informationMachine Learning. Neural Networks
Machine Learning Neural Networks Bryan Pardo, Northwestern University, Machine Learning EECS 349 Fall 2007 Biological Analogy Bryan Pardo, Northwestern University, Machine Learning EECS 349 Fall 2007 THE
More informationLearning and Memory in Neural Networks
Learning and Memory in Neural Networks Guy Billings, Neuroinformatics Doctoral Training Centre, The School of Informatics, The University of Edinburgh, UK. Neural networks consist of computational units
More informationLecture 7 Artificial neural networks: Supervised learning
Lecture 7 Artificial neural networks: Supervised learning Introduction, or how the brain works The neuron as a simple computing element The perceptron Multilayer neural networks Accelerated learning in
More informationComputational Intelligence Lecture 6: Associative Memory
Computational Intelligence Lecture 6: Associative Memory Farzaneh Abdollahi Department of Electrical Engineering Amirkabir University of Technology Fall 2011 Farzaneh Abdollahi Computational Intelligence
More informationHopfield Networks and Boltzmann Machines. Christian Borgelt Artificial Neural Networks and Deep Learning 296
Hopfield Networks and Boltzmann Machines Christian Borgelt Artificial Neural Networks and Deep Learning 296 Hopfield Networks A Hopfield network is a neural network with a graph G = (U,C) that satisfies
More informationHertz, Krogh, Palmer: Introduction to the Theory of Neural Computation. Addison-Wesley Publishing Company (1991). (v ji (1 x i ) + (1 v ji )x i )
Symmetric Networks Hertz, Krogh, Palmer: Introduction to the Theory of Neural Computation. Addison-Wesley Publishing Company (1991). How can we model an associative memory? Let M = {v 1,..., v m } be a
More informationWeek 4: Hopfield Network
Week 4: Hopfield Network Phong Le, Willem Zuidema November 20, 2013 Last week we studied multi-layer perceptron, a neural network in which information is only allowed to transmit in one direction (from
More informationConsider the way we are able to retrieve a pattern from a partial key as in Figure 10 1.
CompNeuroSci Ch 10 September 8, 2004 10 Associative Memory Networks 101 Introductory concepts Consider the way we are able to retrieve a pattern from a partial key as in Figure 10 1 Figure 10 1: A key
More informationChapter 9: The Perceptron
Chapter 9: The Perceptron 9.1 INTRODUCTION At this point in the book, we have completed all of the exercises that we are going to do with the James program. These exercises have shown that distributed
More informationIntroduction to Neural Networks
Introduction to Neural Networks What are (Artificial) Neural Networks? Models of the brain and nervous system Highly parallel Process information much more like the brain than a serial computer Learning
More information18.6 Regression and Classification with Linear Models
18.6 Regression and Classification with Linear Models 352 The hypothesis space of linear functions of continuous-valued inputs has been used for hundreds of years A univariate linear function (a straight
More informationHopfield and Potts-Hopfield Networks
Hopfield and Potts-Hopfield Networks Andy Somogyi Dec 10, 2010 1 Introduction Many early models of neural networks lacked mathematical rigor or analysis. They were, by and large, ad hoc models. John Hopfield
More informationGood vibrations: the issue of optimizing dynamical reservoirs
Good vibrations: the issue of optimizing dynamical reservoirs Workshop on ESNs / LSMs, NIPS 2006 Herbert Jaeger International University Bremen (Jacobs University Bremen, as of Spring 2007) The basic idea:
More informationCSE 5526: Introduction to Neural Networks Hopfield Network for Associative Memory
CSE 5526: Introduction to Neural Networks Hopfield Network for Associative Memory Part VII 1 The basic task Store a set of fundamental memories {ξξ 1, ξξ 2,, ξξ MM } so that, when presented a new pattern
More informationHopfield networks. Lluís A. Belanche Soft Computing Research Group
Lluís A. Belanche belanche@lsi.upc.edu Soft Computing Research Group Dept. de Llenguatges i Sistemes Informàtics (Software department) Universitat Politècnica de Catalunya 2010-2011 Introduction Content-addressable
More informationSections 18.6 and 18.7 Artificial Neural Networks
Sections 18.6 and 18.7 Artificial Neural Networks CS4811 - Artificial Intelligence Nilufer Onder Department of Computer Science Michigan Technological University Outline The brain vs. artifical neural
More informationNeural networks. Chapter 19, Sections 1 5 1
Neural networks Chapter 19, Sections 1 5 Chapter 19, Sections 1 5 1 Outline Brains Neural networks Perceptrons Multilayer perceptrons Applications of neural networks Chapter 19, Sections 1 5 2 Brains 10
More informationNeural networks. Chapter 20. Chapter 20 1
Neural networks Chapter 20 Chapter 20 1 Outline Brains Neural networks Perceptrons Multilayer networks Applications of neural networks Chapter 20 2 Brains 10 11 neurons of > 20 types, 10 14 synapses, 1ms
More informationUsing Variable Threshold to Increase Capacity in a Feedback Neural Network
Using Variable Threshold to Increase Capacity in a Feedback Neural Network Praveen Kuruvada Abstract: The article presents new results on the use of variable thresholds to increase the capacity of a feedback
More informationNeural networks. Chapter 20, Section 5 1
Neural networks Chapter 20, Section 5 Chapter 20, Section 5 Outline Brains Neural networks Perceptrons Multilayer perceptrons Applications of neural networks Chapter 20, Section 5 2 Brains 0 neurons of
More informationNeural Networks. Chapter 18, Section 7. TB Artificial Intelligence. Slides from AIMA 1/ 21
Neural Networks Chapter 8, Section 7 TB Artificial Intelligence Slides from AIMA http://aima.cs.berkeley.edu / 2 Outline Brains Neural networks Perceptrons Multilayer perceptrons Applications of neural
More informationA.I.: Beyond Classical Search
A.I.: Beyond Classical Search Random Sampling Trivial Algorithms Generate a state randomly Random Walk Randomly pick a neighbor of the current state Both algorithms asymptotically complete. Overview Previously
More informationLecture 4: Feed Forward Neural Networks
Lecture 4: Feed Forward Neural Networks Dr. Roman V Belavkin Middlesex University BIS4435 Biological neurons and the brain A Model of A Single Neuron Neurons as data-driven models Neural Networks Training
More informationSections 18.6 and 18.7 Artificial Neural Networks
Sections 18.6 and 18.7 Artificial Neural Networks CS4811 - Artificial Intelligence Nilufer Onder Department of Computer Science Michigan Technological University Outline The brain vs artifical neural networks
More informationEffects of Interactive Function Forms in a Self-Organized Critical Model Based on Neural Networks
Commun. Theor. Phys. (Beijing, China) 40 (2003) pp. 607 613 c International Academic Publishers Vol. 40, No. 5, November 15, 2003 Effects of Interactive Function Forms in a Self-Organized Critical Model
More informationIntroduction Biologically Motivated Crude Model Backpropagation
Introduction Biologically Motivated Crude Model Backpropagation 1 McCulloch-Pitts Neurons In 1943 Warren S. McCulloch, a neuroscientist, and Walter Pitts, a logician, published A logical calculus of the
More informationNeural Networks. Prof. Dr. Rudolf Kruse. Computational Intelligence Group Faculty for Computer Science
Neural Networks Prof. Dr. Rudolf Kruse Computational Intelligence Group Faculty for Computer Science kruse@iws.cs.uni-magdeburg.de Rudolf Kruse Neural Networks 1 Hopfield Networks Rudolf Kruse Neural Networks
More informationMemories Associated with Single Neurons and Proximity Matrices
Memories Associated with Single Neurons and Proximity Matrices Subhash Kak Oklahoma State University, Stillwater Abstract: This paper extends the treatment of single-neuron memories obtained by the use
More informationLearning in State-Space Reinforcement Learning CIS 32
Learning in State-Space Reinforcement Learning CIS 32 Functionalia Syllabus Updated: MIDTERM and REVIEW moved up one day. MIDTERM: Everything through Evolutionary Agents. HW 2 Out - DUE Sunday before the
More informationMachine Learning. Neural Networks. (slides from Domingos, Pardo, others)
Machine Learning Neural Networks (slides from Domingos, Pardo, others) Human Brain Neurons Input-Output Transformation Input Spikes Output Spike Spike (= a brief pulse) (Excitatory Post-Synaptic Potential)
More informationSPSS, University of Texas at Arlington. Topics in Machine Learning-EE 5359 Neural Networks
Topics in Machine Learning-EE 5359 Neural Networks 1 The Perceptron Output: A perceptron is a function that maps D-dimensional vectors to real numbers. For notational convenience, we add a zero-th dimension
More informationIntroduction to Artificial Neural Networks
Facultés Universitaires Notre-Dame de la Paix 27 March 2007 Outline 1 Introduction 2 Fundamentals Biological neuron Artificial neuron Artificial Neural Network Outline 3 Single-layer ANN Perceptron Adaline
More informationMachine Learning. Neural Networks. (slides from Domingos, Pardo, others)
Machine Learning Neural Networks (slides from Domingos, Pardo, others) For this week, Reading Chapter 4: Neural Networks (Mitchell, 1997) See Canvas For subsequent weeks: Scaling Learning Algorithms toward
More informationNeural Networks. CSE 6363 Machine Learning Vassilis Athitsos Computer Science and Engineering Department University of Texas at Arlington
Neural Networks CSE 6363 Machine Learning Vassilis Athitsos Computer Science and Engineering Department University of Texas at Arlington 1 Perceptrons x 0 = 1 x 1 x 2 z = h w T x Output: z x D A perceptron
More informationAssociative Memories (I) Hopfield Networks
Associative Memories (I) Davide Bacciu Dipartimento di Informatica Università di Pisa bacciu@di.unipi.it Applied Brain Science - Computational Neuroscience (CNS) A Pun Associative Memories Introduction
More informationCSC321 Lecture 7: Optimization
CSC321 Lecture 7: Optimization Roger Grosse Roger Grosse CSC321 Lecture 7: Optimization 1 / 25 Overview We ve talked a lot about how to compute gradients. What do we actually do with them? Today s lecture:
More informationNeural Networks Lecture 2:Single Layer Classifiers
Neural Networks Lecture 2:Single Layer Classifiers H.A Talebi Farzaneh Abdollahi Department of Electrical Engineering Amirkabir University of Technology Winter 2011. A. Talebi, Farzaneh Abdollahi Neural
More informationCSC321 Lecture 8: Optimization
CSC321 Lecture 8: Optimization Roger Grosse Roger Grosse CSC321 Lecture 8: Optimization 1 / 26 Overview We ve talked a lot about how to compute gradients. What do we actually do with them? Today s lecture:
More informationLogic Learning in Hopfield Networks
Logic Learning in Hopfield Networks Saratha Sathasivam (Corresponding author) School of Mathematical Sciences, University of Science Malaysia, Penang, Malaysia E-mail: saratha@cs.usm.my Wan Ahmad Tajuddin
More informationArtificial Neural Networks The Introduction
Artificial Neural Networks The Introduction 01001110 01100101 01110101 01110010 01101111 01101110 01101111 01110110 01100001 00100000 01110011 01101011 01110101 01110000 01101001 01101110 01100001 00100000
More informationMachine Learning. Neural Networks. (slides from Domingos, Pardo, others)
Machine Learning Neural Networks (slides from Domingos, Pardo, others) For this week, Reading Chapter 4: Neural Networks (Mitchell, 1997) See Canvas For subsequent weeks: Scaling Learning Algorithms toward
More informationSystems Biology: A Personal View IX. Landscapes. Sitabhra Sinha IMSc Chennai
Systems Biology: A Personal View IX. Landscapes Sitabhra Sinha IMSc Chennai Fitness Landscapes Sewall Wright pioneered the description of how genotype or phenotypic fitness are related in terms of a fitness
More informationOptimization and Gradient Descent
Optimization and Gradient Descent INFO-4604, Applied Machine Learning University of Colorado Boulder September 12, 2017 Prof. Michael Paul Prediction Functions Remember: a prediction function is the function
More informationStorkey Learning Rules for Hopfield Networks
Storey Learning Rules for Hopfield Networs Xiao Hu Bejing Technology Group September 18, 2013 Abstract We summarize the Storey Learning Rules for the Hopfield Model, and evaluate performance relative to
More informationStochastic Networks Variations of the Hopfield model
4 Stochastic Networks 4. Variations of the Hopfield model In the previous chapter we showed that Hopfield networks can be used to provide solutions to combinatorial problems that can be expressed as the
More information6. APPLICATION TO THE TRAVELING SALESMAN PROBLEM
6. Application to the Traveling Salesman Problem 92 6. APPLICATION TO THE TRAVELING SALESMAN PROBLEM The properties that have the most significant influence on the maps constructed by Kohonen s algorithm
More informationSlide10 Haykin Chapter 14: Neurodynamics (3rd Ed. Chapter 13)
Slie10 Haykin Chapter 14: Neuroynamics (3r E. Chapter 13) CPSC 636-600 Instructor: Yoonsuck Choe Spring 2012 Neural Networks with Temporal Behavior Inclusion of feeback gives temporal characteristics to
More informationGrundlagen der Künstlichen Intelligenz
Grundlagen der Künstlichen Intelligenz Neural networks Daniel Hennes 21.01.2018 (WS 2017/18) University Stuttgart - IPVS - Machine Learning & Robotics 1 Today Logistic regression Neural networks Perceptron
More informationCHALMERS, GÖTEBORGS UNIVERSITET. EXAM for ARTIFICIAL NEURAL NETWORKS. COURSE CODES: FFR 135, FIM 720 GU, PhD
CHALMERS, GÖTEBORGS UNIVERSITET EXAM for ARTIFICIAL NEURAL NETWORKS COURSE CODES: FFR 135, FIM 72 GU, PhD Time: Place: Teachers: Allowed material: Not allowed: October 23, 217, at 8 3 12 3 Lindholmen-salar
More informationContent-Addressable Memory Associative Memory Lernmatrix Association Heteroassociation Learning Retrieval Reliability of the answer
Associative Memory Content-Addressable Memory Associative Memory Lernmatrix Association Heteroassociation Learning Retrieval Reliability of the answer Storage Analysis Sparse Coding Implementation on a
More informationClassification: The rest of the story
U NIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN CS598 Machine Learning for Signal Processing Classification: The rest of the story 3 October 2017 Today s lecture Important things we haven t covered yet Fisher
More informationNeural networks: Unsupervised learning
Neural networks: Unsupervised learning 1 Previously The supervised learning paradigm: given example inputs x and target outputs t learning the mapping between them the trained network is supposed to give
More informationThe perceptron learning algorithm is one of the first procedures proposed for learning in neural network models and is mostly credited to Rosenblatt.
1 The perceptron learning algorithm is one of the first procedures proposed for learning in neural network models and is mostly credited to Rosenblatt. The algorithm applies only to single layer models
More informationCOMP 551 Applied Machine Learning Lecture 14: Neural Networks
COMP 551 Applied Machine Learning Lecture 14: Neural Networks Instructor: Ryan Lowe (ryan.lowe@mail.mcgill.ca) Slides mostly by: Class web page: www.cs.mcgill.ca/~hvanho2/comp551 Unless otherwise noted,
More informationAny live cell with less than 2 live neighbours dies. Any live cell with 2 or 3 live neighbours lives on to the next step.
2. Cellular automata, and the SIRS model In this Section we consider an important set of models used in computer simulations, which are called cellular automata (these are very similar to the so-called
More informationSections 18.6 and 18.7 Analysis of Artificial Neural Networks
Sections 18.6 and 18.7 Analysis of Artificial Neural Networks CS4811 - Artificial Intelligence Nilufer Onder Department of Computer Science Michigan Technological University Outline Univariate regression
More informationNeural Networks Lecture 6: Associative Memory II
Neural Networks Lecture 6: Associative Memory II H.A Talebi Farzaneh Abdollahi Department of Electrical Engineering Amirkabir University of Technology Winter 2011. A. Talebi, Farzaneh Abdollahi Neural
More informationIntroduction to Neural Networks
Introduction to Neural Networks Vincent Barra LIMOS, UMR CNRS 6158, Blaise Pascal University, Clermont-Ferrand, FRANCE January 4, 2011 1 / 46 1 INTRODUCTION Introduction History Brain vs. ANN Biological
More information9 Classification. 9.1 Linear Classifiers
9 Classification This topic returns to prediction. Unlike linear regression where we were predicting a numeric value, in this case we are predicting a class: winner or loser, yes or no, rich or poor, positive
More informationarxiv: v2 [nlin.ao] 19 May 2015
Efficient and optimal binary Hopfield associative memory storage using minimum probability flow arxiv:1204.2916v2 [nlin.ao] 19 May 2015 Christopher Hillar Redwood Center for Theoretical Neuroscience University
More informationLast update: October 26, Neural networks. CMSC 421: Section Dana Nau
Last update: October 26, 207 Neural networks CMSC 42: Section 8.7 Dana Nau Outline Applications of neural networks Brains Neural network units Perceptrons Multilayer perceptrons 2 Example Applications
More informationHow to do backpropagation in a brain
How to do backpropagation in a brain Geoffrey Hinton Canadian Institute for Advanced Research & University of Toronto & Google Inc. Prelude I will start with three slides explaining a popular type of deep
More information(Feed-Forward) Neural Networks Dr. Hajira Jabeen, Prof. Jens Lehmann
(Feed-Forward) Neural Networks 2016-12-06 Dr. Hajira Jabeen, Prof. Jens Lehmann Outline In the previous lectures we have learned about tensors and factorization methods. RESCAL is a bilinear model for
More informationError Functions & Linear Regression (1)
Error Functions & Linear Regression (1) John Kelleher & Brian Mac Namee Machine Learning @ DIT Overview 1 Introduction Overview 2 Univariate Linear Regression Linear Regression Analytical Solution Gradient
More informationFinancial Informatics XVII:
Financial Informatics XVII: Unsupervised Learning Khurshid Ahmad, Professor of Computer Science, Department of Computer Science Trinity College, Dublin-, IRELAND November 9 th, 8. https://www.cs.tcd.ie/khurshid.ahmad/teaching.html
More informationARTIFICIAL INTELLIGENCE. Artificial Neural Networks
INFOB2KI 2017-2018 Utrecht University The Netherlands ARTIFICIAL INTELLIGENCE Artificial Neural Networks Lecturer: Silja Renooij These slides are part of the INFOB2KI Course Notes available from www.cs.uu.nl/docs/vakken/b2ki/schema.html
More informationMarkov Chains and MCMC
Markov Chains and MCMC Markov chains Let S = {1, 2,..., N} be a finite set consisting of N states. A Markov chain Y 0, Y 1, Y 2,... is a sequence of random variables, with Y t S for all points in time
More informationPattern Recognition Prof. P. S. Sastry Department of Electronics and Communication Engineering Indian Institute of Science, Bangalore
Pattern Recognition Prof. P. S. Sastry Department of Electronics and Communication Engineering Indian Institute of Science, Bangalore Lecture - 27 Multilayer Feedforward Neural networks with Sigmoidal
More informationSGD and Deep Learning
SGD and Deep Learning Subgradients Lets make the gradient cheating more formal. Recall that the gradient is the slope of the tangent. f(w 1 )+rf(w 1 ) (w w 1 ) Non differentiable case? w 1 Subgradients
More informationArtificial Neural Networks. Q550: Models in Cognitive Science Lecture 5
Artificial Neural Networks Q550: Models in Cognitive Science Lecture 5 "Intelligence is 10 million rules." --Doug Lenat The human brain has about 100 billion neurons. With an estimated average of one thousand
More information3.3 Discrete Hopfield Net An iterative autoassociative net similar to the nets described in the previous sections has been developed by Hopfield
3.3 Discrete Hopfield Net An iterative autoassociative net similar to the nets described in the previous sections has been developed by Hopfield (1982, 1984). - The net is a fully interconnected neural
More informationLinear Regression, Neural Networks, etc.
Linear Regression, Neural Networks, etc. Gradient Descent Many machine learning problems can be cast as optimization problems Define a function that corresponds to learning error. (More on this later)
More informationDynamical Systems and Deep Learning: Overview. Abbas Edalat
Dynamical Systems and Deep Learning: Overview Abbas Edalat Dynamical Systems The notion of a dynamical system includes the following: A phase or state space, which may be continuous, e.g. the real line,
More informationPlan. Perceptron Linear discriminant. Associative memories Hopfield networks Chaotic networks. Multilayer perceptron Backpropagation
Neural Networks Plan Perceptron Linear discriminant Associative memories Hopfield networks Chaotic networks Multilayer perceptron Backpropagation Perceptron Historically, the first neural net Inspired
More informationData Mining Part 5. Prediction
Data Mining Part 5. Prediction 5.5. Spring 2010 Instructor: Dr. Masoud Yaghini Outline How the Brain Works Artificial Neural Networks Simple Computing Elements Feed-Forward Networks Perceptrons (Single-layer,
More informationLearning Long Term Dependencies with Gradient Descent is Difficult
Learning Long Term Dependencies with Gradient Descent is Difficult IEEE Trans. on Neural Networks 1994 Yoshua Bengio, Patrice Simard, Paolo Frasconi Presented by: Matt Grimes, Ayse Naz Erkan Recurrent
More informationNeural Networks 2. 2 Receptive fields and dealing with image inputs
CS 446 Machine Learning Fall 2016 Oct 04, 2016 Neural Networks 2 Professor: Dan Roth Scribe: C. Cheng, C. Cervantes Overview Convolutional Neural Networks Recurrent Neural Networks 1 Introduction There
More information2- AUTOASSOCIATIVE NET - The feedforward autoassociative net considered in this section is a special case of the heteroassociative net.
2- AUTOASSOCIATIVE NET - The feedforward autoassociative net considered in this section is a special case of the heteroassociative net. - For an autoassociative net, the training input and target output
More informationLecture 15: Exploding and Vanishing Gradients
Lecture 15: Exploding and Vanishing Gradients Roger Grosse 1 Introduction Last lecture, we introduced RNNs and saw how to derive the gradients using backprop through time. In principle, this lets us train
More informationMultilayer Perceptron
Outline Hong Chang Institute of Computing Technology, Chinese Academy of Sciences Machine Learning Methods (Fall 2012) Outline Outline I 1 Introduction 2 Single Perceptron 3 Boolean Function Learning 4
More informationArtificial Neural Networks. Historical description
Artificial Neural Networks Historical description Victor G. Lopez 1 / 23 Artificial Neural Networks (ANN) An artificial neural network is a computational model that attempts to emulate the functions of
More informationLecture - 24 Radial Basis Function Networks: Cover s Theorem
Neural Network and Applications Prof. S. Sengupta Department of Electronic and Electrical Communication Engineering Indian Institute of Technology, Kharagpur Lecture - 24 Radial Basis Function Networks:
More information