Dynamics of Ordinary and Recurrent Hopfield Networks: Novel Themes
|
|
- Philippa Horn
- 5 years ago
- Views:
Transcription
1 2017 IEEE 7th International Advance Computing Conference Dynamics of Ordinary and Recurrent Hopfield Networks: Novel Themes Siva Prasad Raju Bairaju, Rama Murthy Garimella, Ayush Jha,Anil Rayala APIIIT RGUKT RKValley International Institute of Information Technology,Hyderabad Indian Institute of Technology, Guwahati Abstract In this research paper several interesting themes related to Hopfield network are explored and concrete results are derived For instance, convergence in partial parallel mode of operation, convergence when synaptic weight matrix, W is row diagonally dominant case Also, structured recurrent Hopfield network with companion matrix as W is studied The dynamics when the activation function is a step function, the state space is asymmetric hypercube are also studied Finally several experimental results are derived Index Terms Structured recurrent Hopfield network, asymmetric hypercube, activation function, companion matrix Dynamics with synaptic matrix as a Row Diagonal Dominant Matrix and Companion matrix are discussed in section IV and V respectively Dynamics in case of step function as an activation function is discussed in section VI and Experimental results with various weight synaptic matrices is discussed in section VII Applications of the proposed Structured Recurrent Hopfield Network are discussed in section VIII Future Work is discussed in section IX The paper concludes in section X II RECURRENT HOPFIELD NEURAL NETWORK I INTRODUCTION Hopfield proposed an artificial neural network(ann) which acts as an associative memory This neural network was subjected to intensive research and several results are reported But certain research issues related to such an ANN remain unresolved For instance, convergence issues related to partial(semi) parallel modes of operation remain unexplored In this research paper we address this problem Also dynamics of Hopfield neural network(hnn) whose synaptic weight matrix is diagonally dominant is explored in this paper RHNN is also a ANN The fundamental feature of a recurrent neural network is that network contains at least one feedback connection, so the activations can flow round in a loop As shown in the fig 1 output of one neuron will act as the input for another neuron Thus, connections in the network can circulate continuously Fig 1 Recurrent Hopfield Network Dynamics of ANN (in the spirit of Hopfield Neural Network) whose synaptic weight matrix is asymmetric is also addressed only recently [2] Such a neural network, called Recurrent Hopfield Neural Network(RHNN) was subjected to experimental investigation In this research paper a structured RHNN whose synaptic weight matrix is a companion matrix is studied Several other issues related to HNN are explored If weight matrix is symmetric then we call it as Ordinary Hopfield Network and when it is asymmetric then we call it as Recurrent Hopfield Network This research paper is designed as follows: In section II, Recurrent Hopfield Neural Network is introduced Convergence in partial parallel mode operation is discussed in section III It is a dynamical system which is non-linear and is represented by a weighted, directed graph The nodes of the graph repre /17 $ IEEE 6 DOI /IACC20179
2 sent neurons and the edges of the graph represent the weights Each neuron contains a activation (threshold) value Thus, an RHNN can be represented by a thresholding vector T and a synaptic weight matrix W Each neuron assumes a state which lies in {-1,1} The number of neurons represents the order of the network Let the state of the i th neuron at time t be denoted by V i (t) Thus the state of the non-linear dynamical system having N neurons is represented by N 1 vector, V(t) The state updation at the i th node is represented by the following equation, N V i (t +1)=sign[ W ij V j (t) T i ] j=1 where sign is equal to signum function defined as { +1 if t 0 sign(t) = 1 if t<0 Depending on the state updation, the Recurrent hopfield network operation can be in the following modes of operation Serial Mode: If the updation takes place on only one neuron at an instant of time then this type of updation is said to be Serial mode Fully Parallel Mode: If the state updation takes place at all neurons at a time instant, it is called the Fully Parallel Mode Partial(Semi) Parallel Mode: If the state updation takes place simultaneously on only some selected neurons at a time instant, then the network is said to be Partial(Semi) Parallel Mode Remark: Dynamics of Recurrent Hopfield Neural Network(RHNN) is exactly same as that of Ordinary Hopfield Neural Network(OHNN)if and only if the matrix W is symmetric For the cause of integrity, We now include the dynamics of OHNN with W being a symmetric matrix In the state space of (OHNN),a non-linear dynamical system, there are certain well-known states, called the stable states Definition: A state, V (t) is called a stable state if and only if V (t) =sign[w V (t) T ] Thus, if the HNN is in its stable state then the mode of operation cannot affect the HNN Thus, there is no any change in HNN when a stable state is reached III CONVERGENCE IN PARTIAL PARALLEL MODE Theorem 1 If a Hopfield network with symmetric matrix as synaptic matrix is operated in partial(semi) parallel mode, then the network either converges to a stable state or a cycle of length 2 is achieved, after finite number of iterations Proof: The proof is similar to the proof presented in [1] Let R=(M,T) be our original network with n number of nodes/neurons and let us assume that a new network ˆR= ( ˆM, ˆT), with M = [ ] 0 M, T = M 0 [ ] T T The new network R is a hopfield network with 2n number of nodes and also bipartite graph due to our construction The total set of neurons in R can be divided into two disjoint sets, let them be D1 and D2 such that no two neurons in the same set has a non zero weight It is possible because, R is a bipartite graph We will also find that for every neuron 2 D1, there exists a similar neuron 2 D2 which has same edge set There exists a serial mode of operation in R which is equivalent to a partial mode of operation in R Now, we will look into the details let V 0 be the initial state of R and (V 0,V 0 ) be the inital state of R Now, clearly, partial parallel updates in R is equivalent to partial parallel update in D1 and partial parallel update in D2 and proceeding with different possible orders, in the sense that state of R is equal to either state of D1 or D2 depending upon which set, the last evaluation is performed Now, we know there are no internal edge connections in D1 or D2, which implies that, partial parallel update in D1 or D2 is equal to a serial update Hence for every partial parallel update in R, there exist equivalent serial mode updates in R Since we already know that serial mode in a Hopfield network with symmetric synaptic matrix converges to a stable state, that implies, we reach a state state in R Finally, state of R is equal to either state of D1 or D2 depending on last evaluation So, if D1 and D2 are equal, we end up with a stable state for R otherwise we get a cycle of length 2 The Theorem is also empirically evaluated IV DYNAMICS IN CASE OF ROW DIAGONAL DOMINANT MATRIX In this section we study the dynamics of Special case of Recurrent Hopfield Network for fully parallel mode of operation where the synaptic matrix W is a strictly diagonally dominant matrix ie each diagonal element W ii is larger in magnitude than the sum of the magnitudes of all other elements in the i th row W ij < W ii j=1,j i Some of the properties of a diagonal matrix are: 7
3 It is nonsingular Gaussian elimination can be applied to it without performing any row interchanges Theorem 2 If a Hopfield network with row diagonally dominant matrix as synaptic weight matrix is operated in fully parallel mode, then the network either converges to stable state or a cycle of length 2 Proof: Case 1: All the diagonal elements are positive Here, if the network is operated in parallel update mode, then convergence is observed Let the initial state of the network be V 1 and weight matrix be W Now, due to the structure of our W (row dominant matrix with positive diagonal elements), we can say that, even after a state update, the state of the network will be V 1 which means that network reached a stable state The structure of W ensures that stable state is reached in one step for some initial conditions For other initial conditions, cycle of length 2 is reached Ex: V 1 = [ ] T Here V 1 be the initial state of the network Then, it converges to stable state that is V 1 = [ ] T The above reasoning can be generalized to any case because of the structure of our synaptic matrix W Case 2: All the diagonal elements are negative Let the initial state of the network be V 1 and weight matrix be W Now, due to the structure of our W (row dominant matrix with negative diagonal elements), we can say that, even after a state update, the state of the network will be V 1 that means it is a stable state V 1 = [ ] T Here V 1 be the initial state of the network Then, it will converge to cycle of length 2 (ie, cycles of the state space are of length 2 ) If we start update from state V 1 = [ ] T then it will go to state V 2 = [ ] T and then it will oscillate between two states The above reasoning can be generalized to any case because of the structure of our synaptic matrix W Case 3: Rephrase some diagonal elements are positive and some are negative Let the initial state of the network be V 1 and weight matrix be W Now, due to the structure of our W (row dominant matrix with some negative, some positive diagonal elements), we can say that, even after a state update, the state of the network will be V 1 which means the network reached a stable state V 1 = [ ] T Here V 1 be the initial state of the network Then, it converges to cycle of length 2 (ie, cycles of the state space are of length 2) If we start update from state V 1 = [ ] T then it will go to state V 2 = [ ] T and then it will oscillate between two states The above reasoning can be generalized to any case because of the structure of our synaptic matrix W V INTERESTING HOPFIELD NETWORK In [2] and [3], authors proposed novel homogeneous recurrent networks based on modulo neuron and ideas related to linear congruential sequence respectively So, we now propose an interesting Hopfield network with the above motivations Consider a discrete time sequence generated by the following non linear dynamical system x(n + M) =sign({a 1 x(n) +a 2 x(n +1)+ + a n x(n + M 1)}) where a is are real numbers Now we represent such a system by choosing an appropriate state vector Let Y be defined as x(n) x(n +1) Y [n] = x(n + M 1) be the state vector associated with our proposed discrete time non-linear dynamical system Let the initial vector x(0) x(1) Y [0] = x(m 1) 8
4 lie on the unit hypercube Now, with the above choice of initial vector, the vector-matrix difference equation associated with above equation is Y (n +1)=sign(W Y (n)) for n 0 In the above non linear difference equation, the matrix W is given by a M a M-1 a M-2 a 1 Thus the state transition matrix of our non linear dynamical system is a companion matrix which is interesting Definition:A state, V (t) is called anti-stable state, if and only if V (t) = sign(wv(t)) By the properties of W, if it has a eigenvalue λ then the corresponding eigenvector will be of the form: 1 λ X = λ (n 1) Theorem 3 Recurrent Hopfield network based on Companion asymmetric matrix can have at most two programmed stable states and two programmed anti-stable states on the hypercube Proof: For a companion matrix based recurrent Hopfield network, we can choose 2 states (that are eigenvectors corresponding to positive eigenvalue, on the hypercube as stable state) For a companion matrix based recurrent Hopfield network, we can choose 2 states (that are eigenvectors corresponding to negative eigenvalue, on the hypercube as anti-stable state) For the eigenvector X to lie on unit hypercube λ must be {+1 or -1} Thus corresponding to eigenvalue +1 the following can be stable states : [ ] T,[ 1 1 1] T Similarly corresponding to eigenvalue -1, the following can be anti-stable states: [ ] T, [ ] T From above theorem, and the property of A, the only possible values for λ is 1 or -1 to have a valid state Here matrix A = matrix W, Case 1: Suppose λ = 1, then we will get: A X=λ X Now applying signum function on both sides sign(a X) = sign (λ X)=X that is also how we define a stable state The question here is that if X is a stable state, then what about -X? For this thing consider Z =-X then to get the next state sign(a Z) = sign (A -X) = -sign (A X)=-X which again is the definition of stable state So for eigenvalue λ = 1, we get 2 stable statesthat is X, -X Case 2: Suppose it has -1 as a eigenvalue, with corresponding eigenvector lying on the hypercube(from the above property) Then we get: sign(a X) = sign( λ X) = sign(-x) = -X ie,sign(a X)=-X which is the definition of anti-stable states So for eigenvalue λ = -1 we get anti-stable stateswe chose 1 and -1 because these are the only eigenvalues whose corresponding eigenvector is a possible state Experimental results: 1) Structured recurrent Hopfield network: If network is operating in fully parallel mode and A is asymmetric companion & stochastic matrix then the cycle length of the network is bounded which means that the cycle length is a small number Number of instances L 1 L 2 3 L ) Structured recurrent Hopfield network: If the network is operating in fully parallel mode and A is asymmetric companion & rows sum to 2, 5 (>1) Then the network is Number of instances RowsSum L VI ACTIVATION FUNCTION The activation function converts the activation level of a node into an output signal Various types of activation functions are available in use with ANNs Unit step function is one of the activation functions That is { +1 if t 0 step(t) = 0 if t<0 9
5 If we take this unit step function as a activation function then the following theorems hold 1) If a Hopfield network with symmetric matrix is operated in fully parallel mode, then the network either converges to a stable state or a cycle of length 2 2) If a Hopfield network with arbitrary symmetric matrix is operated in partial(semi) parallel mode, then the network either converges to a stable state or a cycle of length 2 3) If a Hopfield network with asymmetric matrix with positive elements is operated in fully parallel mode, then the network converge to stable state 4)Let N =(W,T)be the neural network If N is operating in partial parallel mode, W is asymmetric matrix with positive elements then the network will converge to a stable state These are all trivial cases VII RECURRENT HOPFIELD NETWORK BASED ON SPECIAL WEIGHT MATRICES: EXPERIMENTAL RESULTS 1) Asymmetric hypercube: symmetric synaptic weight matrix If the network is operating in fully parallel mode, A is asymmetric hypercube symmetric synaptic weight matrix then the network will converge to a stable state or to a cycle of length 2 2) If we take non-negative diagonal asymmetric matrix and the network s operation is in fully parallel mode then cycle length of the network is bounded which means that the cycle length is a small number Number of instances L 1 2 L IX FUTURE WORK Significance of Companion matrix based recurrent Hopfield network Other structured synaptic weight matrices: convergence results Practical applications of recurrent Hopfield neural network (RHNNs) X CONCLUSION In this research paper, convergence in partial parallel mode of operation is discussed An interesting structured recurrent Hopfield network(with companion synaptic weight matrix) is proposed Convergence when synaptic weight matrix, W is row diagonally dominant case is discussed The dynamics when the activation function is a step function are discussed Experimental results for different types of recurrent Hopfield networks are shown Some applications are proposed REFERENCES [1] Jehoshua Bruck and Joseph W Goodman, A Generalized Convergence Theorem for Neural Networks, IEEE TRANSACTIONS ON INFORMA- TION THEORY, VOL 34, NO 5, SEPTEMBER 1988 [2] G Rama Murthy, Berkay, Moncef Gabbouji, On the dynamics of a recurrent Hopfield Network,International Joint Conference on Neural Networks(IJCNN-2015),July 2015 [3] G Rama Murthy, Multi-dimensional Neural Networks:Unified Theory,New Age International Publishers,New Delhi,2007 [4] JJ Hopfield, Neural Networks and Physical systems with Emergent Collective Computational Abilities, Proceedings of National Academy of Sciences, USA Vol 79,pp , 1982 [5] ZONG-BEN Xu,Guo-QING Hu AND CHUNG-PING KWONG,Asymmetric Hopfield-type Networks: Theory and Applications, 1996 Elsevier Science LtdVol 9, No 3, pp , 1996 [6] E Goles, Antisymmetrical neural networks, Discrete ApplMath,vol 13,pp97-100,1986 [7] Hwa-Long Gau, Pei Yuan Wub, Companion matrices: reducibility, numerical ranges and similarity to contractions, ELSEVIER, Linear Algebra and its applications 383(2004) [8] Donq-Liang Lee, New Stability Conditions for Hopfield Networks in Partial Simultaneous Update Mode, IEEE TRANSACTIONS ON NEURAL NETWORKS VOL 10, NO4,JULY 1999 VIII APPLICATIONS Neural Network such as Hopfield Network have diverse applications Generally they are use for, associative memory: the network is able to memorize some states, patterns Traditionally, associative memories are defined by associating a single state (ie stable state) with the noise corrupted versions of it In [2], the authors proposed the concept of multi-state associative memories by associating a initial condition with a cycle of states 10
Dynamics of Structured Complex Recurrent Hopfield Networks
Dynamics of Structured Complex Recurrent Hopfield Networks by Garimella Ramamurthy Report No: IIIT/TR/2016/-1 Centre for Security, Theory and Algorithms International Institute of Information Technology
More informationOn the Dynamics of a Recurrent Hopfield Network
On the Dynamics of a Recurrent Hopfield Network Rama Garimella Department of Signal Processing Tampere University of Technology Tampere, Finland Email: rammurthy@iiit.ac.in Berkay Kicanaoglu Department
More informationHopfield-Amari Neural Network:
Hopfield-Amari Neural Network: Minimization of Quadratic Forms Dr G Rama Murthy PhD Associate Professor IIIT Hyderabad Andhra Pradesh, India E-mail: rammurthy@iiitacin B Nischal(BTech) NIT Hamirpur Himachal
More informationConvolutional Associative Memory: FIR Filter Model of Synapse
Convolutional Associative Memory: FIR Filter Model of Synapse Rama Murthy Garimella 1, Sai Dileep Munugoti 2, Anil Rayala 1 1 International Institute of Information technology, Hyderabad, India. rammurthy@iiit.ac.in,
More informationOn the Dynamics of a Recurrent Hopfield Network
On the Dynamics of a Recurrent Hopfield Network by Garimella Ramamurthy Report No: III/R/2015/-1 Centre for Communications International Institute of Information echnology Hyderabad - 500 032, INDIA April
More informationOptimization of Quadratic Forms: NP Hard Problems : Neural Networks
1 Optimization of Quadratic Forms: NP Hard Problems : Neural Networks Garimella Rama Murthy, Associate Professor, International Institute of Information Technology, Gachibowli, HYDERABAD, AP, INDIA ABSTRACT
More informationOptimization of Quadratic Forms: Unified Minimum/Maximum Cut Computation in Directed/Undirected Graphs
Optimization of Quadratic Forms: Unified Minimum/Maximum Cut Computation in Directed/Undirected Graphs by Garimella Ramamurthy Report No: IIIT/TR/2015/-1 Centre for Security, Theory and Algorithms International
More informationHopfield Neural Network
Lecture 4 Hopfield Neural Network Hopfield Neural Network A Hopfield net is a form of recurrent artificial neural network invented by John Hopfield. Hopfield nets serve as content-addressable memory systems
More informationArtificial Intelligence Hopfield Networks
Artificial Intelligence Hopfield Networks Andrea Torsello Network Topologies Single Layer Recurrent Network Bidirectional Symmetric Connection Binary / Continuous Units Associative Memory Optimization
More informationConsider the way we are able to retrieve a pattern from a partial key as in Figure 10 1.
CompNeuroSci Ch 10 September 8, 2004 10 Associative Memory Networks 101 Introductory concepts Consider the way we are able to retrieve a pattern from a partial key as in Figure 10 1 Figure 10 1: A key
More informationShort Term Memory Quantifications in Input-Driven Linear Dynamical Systems
Short Term Memory Quantifications in Input-Driven Linear Dynamical Systems Peter Tiňo and Ali Rodan School of Computer Science, The University of Birmingham Birmingham B15 2TT, United Kingdom E-mail: {P.Tino,
More informationCHAPTER 3. Pattern Association. Neural Networks
CHAPTER 3 Pattern Association Neural Networks Pattern Association learning is the process of forming associations between related patterns. The patterns we associate together may be of the same type or
More informationAlgebraic Constructions of Graphs
Spectral Graph Theory Lecture 15 Algebraic Constructions of Graphs Daniel A. Spielman October 17, 2012 15.1 Overview In this lecture, I will explain how to make graphs from linear error-correcting codes.
More informationThe Hypercube Graph and the Inhibitory Hypercube Network
The Hypercube Graph and the Inhibitory Hypercube Network Michael Cook mcook@forstmannleff.com William J. Wolfe Professor of Computer Science California State University Channel Islands william.wolfe@csuci.edu
More informationA Cross-Associative Neural Network for SVD of Nonsquared Data Matrix in Signal Processing
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 12, NO. 5, SEPTEMBER 2001 1215 A Cross-Associative Neural Network for SVD of Nonsquared Data Matrix in Signal Processing Da-Zheng Feng, Zheng Bao, Xian-Da Zhang
More informationAssignment # 4. K-WTA neural networks are relatively simple. N neurons, with all the connection strengths equal to -1 (with no self-connections).
Farrukh Jabeen COM 58 Mathematical Methods In AI Assignment # 4 This problem asks you to analyze a K-WTA network (K winner take all). First read the K-WTA reference article, then answer the questions below.
More informationIntroduction to Neural Networks
Introduction to Neural Networks What are (Artificial) Neural Networks? Models of the brain and nervous system Highly parallel Process information much more like the brain than a serial computer Learning
More informationOn the SIR s ( Signal -to- Interference -Ratio) in. Discrete-Time Autonomous Linear Networks
On the SIR s ( Signal -to- Interference -Ratio) in arxiv:93.9v [physics.data-an] 9 Mar 9 Discrete-Time Autonomous Linear Networks Zekeriya Uykan Abstract In this letter, we improve the results in [5] by
More informationIterative Encoding of Low-Density Parity-Check Codes
Iterative Encoding of Low-Density Parity-Check Codes David Haley, Alex Grant and John Buetefuer Institute for Telecommunications Research University of South Australia Mawson Lakes Blvd Mawson Lakes SA
More informationNumerical Methods - Numerical Linear Algebra
Numerical Methods - Numerical Linear Algebra Y. K. Goh Universiti Tunku Abdul Rahman 2013 Y. K. Goh (UTAR) Numerical Methods - Numerical Linear Algebra I 2013 1 / 62 Outline 1 Motivation 2 Solving Linear
More informationLIMITING PROBABILITY TRANSITION MATRIX OF A CONDENSED FIBONACCI TREE
International Journal of Applied Mathematics Volume 31 No. 18, 41-49 ISSN: 1311-178 (printed version); ISSN: 1314-86 (on-line version) doi: http://dx.doi.org/1.173/ijam.v31i.6 LIMITING PROBABILITY TRANSITION
More informationStudy Notes on Matrices & Determinants for GATE 2017
Study Notes on Matrices & Determinants for GATE 2017 Matrices and Determinates are undoubtedly one of the most scoring and high yielding topics in GATE. At least 3-4 questions are always anticipated from
More informationRemark By definition, an eigenvector must be a nonzero vector, but eigenvalue could be zero.
Sec 6 Eigenvalues and Eigenvectors Definition An eigenvector of an n n matrix A is a nonzero vector x such that A x λ x for some scalar λ A scalar λ is called an eigenvalue of A if there is a nontrivial
More informationhttp://www.math.uah.edu/stat/markov/.xhtml 1 of 9 7/16/2009 7:20 AM Virtual Laboratories > 16. Markov Chains > 1 2 3 4 5 6 7 8 9 10 11 12 1. A Markov process is a random process in which the future is
More informationLinear Algebra. Matrices Operations. Consider, for example, a system of equations such as x + 2y z + 4w = 0, 3x 4y + 2z 6w = 0, x 3y 2z + w = 0.
Matrices Operations Linear Algebra Consider, for example, a system of equations such as x + 2y z + 4w = 0, 3x 4y + 2z 6w = 0, x 3y 2z + w = 0 The rectangular array 1 2 1 4 3 4 2 6 1 3 2 1 in which the
More informationContents. Preface... xi. Introduction...
Contents Preface... xi Introduction... xv Chapter 1. Computer Architectures... 1 1.1. Different types of parallelism... 1 1.1.1. Overlap, concurrency and parallelism... 1 1.1.2. Temporal and spatial parallelism
More informationIN THIS PAPER, we consider a class of continuous-time recurrent
IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS II: EXPRESS BRIEFS, VOL. 51, NO. 4, APRIL 2004 161 Global Output Convergence of a Class of Continuous-Time Recurrent Neural Networks With Time-Varying Thresholds
More informationEigenvalues and Eigenvectors
5 Eigenvalues and Eigenvectors 5.2 THE CHARACTERISTIC EQUATION DETERMINANATS n n Let A be an matrix, let U be any echelon form obtained from A by row replacements and row interchanges (without scaling),
More informationChapter Two Elements of Linear Algebra
Chapter Two Elements of Linear Algebra Previously, in chapter one, we have considered single first order differential equations involving a single unknown function. In the next chapter we will begin to
More informationComputer Vision Group Prof. Daniel Cremers. 11. Sampling Methods: Markov Chain Monte Carlo
Group Prof. Daniel Cremers 11. Sampling Methods: Markov Chain Monte Carlo Markov Chain Monte Carlo In high-dimensional spaces, rejection sampling and importance sampling are very inefficient An alternative
More informationUsing Variable Threshold to Increase Capacity in a Feedback Neural Network
Using Variable Threshold to Increase Capacity in a Feedback Neural Network Praveen Kuruvada Abstract: The article presents new results on the use of variable thresholds to increase the capacity of a feedback
More informationScientific Computing
Scientific Computing Direct solution methods Martin van Gijzen Delft University of Technology October 3, 2018 1 Program October 3 Matrix norms LU decomposition Basic algorithm Cost Stability Pivoting Pivoting
More informationLinear Algebra review Powers of a diagonalizable matrix Spectral decomposition
Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition Prof. Tesler Math 283 Fall 2018 Also see the separate version of this with Matlab and R commands. Prof. Tesler Diagonalizing
More informationReview of Linear Algebra
Review of Linear Algebra Definitions An m n (read "m by n") matrix, is a rectangular array of entries, where m is the number of rows and n the number of columns. 2 Definitions (Con t) A is square if m=
More informationMTH 215: Introduction to Linear Algebra
MTH 215: Introduction to Linear Algebra Lecture 5 Jonathan A. Chávez Casillas 1 1 University of Rhode Island Department of Mathematics September 20, 2017 1 LU Factorization 2 3 4 Triangular Matrices Definition
More informationPattern Association or Associative Networks. Jugal Kalita University of Colorado at Colorado Springs
Pattern Association or Associative Networks Jugal Kalita University of Colorado at Colorado Springs To an extent, learning is forming associations. Human memory associates similar items, contrary/opposite
More informationConvolutional networks. Sebastian Seung
Convolutional networks Sebastian Seung Convolutional network Neural network with spatial organization every neuron has a location usually on a grid Translation invariance synaptic strength depends on locations
More informationLecture 4: Gaussian Elimination and Homogeneous Equations
Lecture 4: Gaussian Elimination and Homogeneous Equations Reduced Row Echelon Form An augmented matrix associated to a system of linear equations is said to be in Reduced Row Echelon Form (RREF) if the
More informationFundamentals of Matrices
Maschinelles Lernen II Fundamentals of Matrices Christoph Sawade/Niels Landwehr/Blaine Nelson Tobias Scheffer Matrix Examples Recap: Data Linear Model: f i x = w i T x Let X = x x n be the data matrix
More informationNumerical Analysis: Solutions of System of. Linear Equation. Natasha S. Sharma, PhD
Mathematical Question we are interested in answering numerically How to solve the following linear system for x Ax = b? where A is an n n invertible matrix and b is vector of length n. Notation: x denote
More informationApplication of hopfield network in improvement of fingerprint recognition process Mahmoud Alborzi 1, Abbas Toloie- Eshlaghy 1 and Dena Bazazian 2
5797 Available online at www.elixirjournal.org Computer Science and Engineering Elixir Comp. Sci. & Engg. 41 (211) 5797-582 Application hopfield network in improvement recognition process Mahmoud Alborzi
More informationDM559 Linear and Integer Programming. Lecture 2 Systems of Linear Equations. Marco Chiarandini
DM559 Linear and Integer Programming Lecture Marco Chiarandini Department of Mathematics & Computer Science University of Southern Denmark Outline 1. Outline 1. 3 A Motivating Example You are organizing
More informationSolution of Linear Equations
Solution of Linear Equations (Com S 477/577 Notes) Yan-Bin Jia Sep 7, 07 We have discussed general methods for solving arbitrary equations, and looked at the special class of polynomial equations A subclass
More informationEEE 241: Linear Systems
EEE 4: Linear Systems Summary # 3: Introduction to artificial neural networks DISTRIBUTED REPRESENTATION An ANN consists of simple processing units communicating with each other. The basic elements of
More informationModeling and Stability Analysis of a Communication Network System
Modeling and Stability Analysis of a Communication Network System Zvi Retchkiman Königsberg Instituto Politecnico Nacional e-mail: mzvi@cic.ipn.mx Abstract In this work, the modeling and stability problem
More informationMarkov Chains and Spectral Clustering
Markov Chains and Spectral Clustering Ning Liu 1,2 and William J. Stewart 1,3 1 Department of Computer Science North Carolina State University, Raleigh, NC 27695-8206, USA. 2 nliu@ncsu.edu, 3 billy@ncsu.edu
More informationRemark 1 By definition, an eigenvector must be a nonzero vector, but eigenvalue could be zero.
Sec 5 Eigenvectors and Eigenvalues In this chapter, vector means column vector Definition An eigenvector of an n n matrix A is a nonzero vector x such that A x λ x for some scalar λ A scalar λ is called
More informationLinear Algebra review Powers of a diagonalizable matrix Spectral decomposition
Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition Prof. Tesler Math 283 Fall 2016 Also see the separate version of this with Matlab and R commands. Prof. Tesler Diagonalizing
More informationAverage-Consensus of Multi-Agent Systems with Direct Topology Based on Event-Triggered Control
Outline Background Preliminaries Consensus Numerical simulations Conclusions Average-Consensus of Multi-Agent Systems with Direct Topology Based on Event-Triggered Control Email: lzhx@nankai.edu.cn, chenzq@nankai.edu.cn
More informationIn biological terms, memory refers to the ability of neural systems to store activity patterns and later recall them when required.
In biological terms, memory refers to the ability of neural systems to store activity patterns and later recall them when required. In humans, association is known to be a prominent feature of memory.
More informationNeural Nets in PR. Pattern Recognition XII. Michal Haindl. Outline. Neural Nets in PR 2
Neural Nets in PR NM P F Outline Motivation: Pattern Recognition XII human brain study complex cognitive tasks Michal Haindl Faculty of Information Technology, KTI Czech Technical University in Prague
More information18.6 Regression and Classification with Linear Models
18.6 Regression and Classification with Linear Models 352 The hypothesis space of linear functions of continuous-valued inputs has been used for hundreds of years A univariate linear function (a straight
More informationOn the Hopfield algorithm. Foundations and examples
General Mathematics Vol. 13, No. 2 (2005), 35 50 On the Hopfield algorithm. Foundations and examples Nicolae Popoviciu and Mioara Boncuţ Dedicated to Professor Dumitru Acu on his 60th birthday Abstract
More informationChapter 5. Linear Algebra. A linear (algebraic) equation in. unknowns, x 1, x 2,..., x n, is. an equation of the form
Chapter 5. Linear Algebra A linear (algebraic) equation in n unknowns, x 1, x 2,..., x n, is an equation of the form a 1 x 1 + a 2 x 2 + + a n x n = b where a 1, a 2,..., a n and b are real numbers. 1
More informationLecture 7 Artificial neural networks: Supervised learning
Lecture 7 Artificial neural networks: Supervised learning Introduction, or how the brain works The neuron as a simple computing element The perceptron Multilayer neural networks Accelerated learning in
More informationMarkov Chains, Random Walks on Graphs, and the Laplacian
Markov Chains, Random Walks on Graphs, and the Laplacian CMPSCI 791BB: Advanced ML Sridhar Mahadevan Random Walks! There is significant interest in the problem of random walks! Markov chain analysis! Computer
More informationMIT Final Exam Solutions, Spring 2017
MIT 8.6 Final Exam Solutions, Spring 7 Problem : For some real matrix A, the following vectors form a basis for its column space and null space: C(A) = span,, N(A) = span,,. (a) What is the size m n of
More informationEigenvalues and Eigenvectors
5 Eigenvalues and Eigenvectors 5.2 THE CHARACTERISTIC EQUATION DETERMINANATS nn Let A be an matrix, let U be any echelon form obtained from A by row replacements and row interchanges (without scaling),
More informationChapter 3. Linear and Nonlinear Systems
59 An expert is someone who knows some of the worst mistakes that can be made in his subject, and how to avoid them Werner Heisenberg (1901-1976) Chapter 3 Linear and Nonlinear Systems In this chapter
More informationInteractive Interference Alignment
Interactive Interference Alignment Quan Geng, Sreeram annan, and Pramod Viswanath Coordinated Science Laboratory and Dept. of ECE University of Illinois, Urbana-Champaign, IL 61801 Email: {geng5, kannan1,
More informationSection 9.2: Matrices. Definition: A matrix A consists of a rectangular array of numbers, or elements, arranged in m rows and n columns.
Section 9.2: Matrices Definition: A matrix A consists of a rectangular array of numbers, or elements, arranged in m rows and n columns. That is, a 11 a 12 a 1n a 21 a 22 a 2n A =...... a m1 a m2 a mn A
More informationIntrinsic products and factorizations of matrices
Available online at www.sciencedirect.com Linear Algebra and its Applications 428 (2008) 5 3 www.elsevier.com/locate/laa Intrinsic products and factorizations of matrices Miroslav Fiedler Academy of Sciences
More informationMATRICES. knowledge on matrices Knowledge on matrix operations. Matrix as a tool of solving linear equations with two or three unknowns.
MATRICES After studying this chapter you will acquire the skills in knowledge on matrices Knowledge on matrix operations. Matrix as a tool of solving linear equations with two or three unknowns. List of
More informationLecture 1: Systems of linear equations and their solutions
Lecture 1: Systems of linear equations and their solutions Course overview Topics to be covered this semester: Systems of linear equations and Gaussian elimination: Solving linear equations and applications
More informationLinear Algebra. The analysis of many models in the social sciences reduces to the study of systems of equations.
POLI 7 - Mathematical and Statistical Foundations Prof S Saiegh Fall Lecture Notes - Class 4 October 4, Linear Algebra The analysis of many models in the social sciences reduces to the study of systems
More informationSCRAM: Statistically Converging Recurrent Associative Memory
SCRAM: Statistically Converging Recurrent Associative Memory Sylvain Chartier Sébastien Hélie Mounir Boukadoum Robert Proulx Department of computer Department of computer science, UQAM, Montreal, science,
More informationValiant s Neuroidal Model
Valiant s Neuroidal Model Leslie Valiant CSE Department, UCSD Presented by: Max Hopkins November 13, 2018 Leslie Valiant (Harvard University) November 13, 2018 1 / 15 Roadmap 1 Modeling Memory and Association
More informationASSIGNMENT BOOKLET. Bachelor's Degree Programme LINEAR ALGEBRA. It is compulsory to submit the assignment before filling in the exam form.
ASSIGNMENT BOOKLET MTE- Bachelor's Degree Programme LINEAR ALGEBRA (Valid from st January, to st December, ) It is compulsory to submit the assignment before filling in the exam form. School of Sciences
More information3.3 Discrete Hopfield Net An iterative autoassociative net similar to the nets described in the previous sections has been developed by Hopfield
3.3 Discrete Hopfield Net An iterative autoassociative net similar to the nets described in the previous sections has been developed by Hopfield (1982, 1984). - The net is a fully interconnected neural
More informationHOPFIELD neural networks (HNNs) are a class of nonlinear
IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS II: EXPRESS BRIEFS, VOL. 52, NO. 4, APRIL 2005 213 Stochastic Noise Process Enhancement of Hopfield Neural Networks Vladimir Pavlović, Member, IEEE, Dan Schonfeld,
More informationFACULTY OF MATHEMATICAL STUDIES MATHEMATICS FOR PART I ENGINEERING. Lectures
FACULTY OF MATHEMATICAL STUDIES MATHEMATICS FOR PART I ENGINEERING Lectures MODULE 8 MATRICES III Rank of a matrix 2 General systems of linear equations 3 Eigenvalues and eigenvectors Rank of a matrix
More informationStructured Multi Matrix Variate, Matrix Polynomial Equations: Solution Techniques
Structured Multi Matrix Variate, Matrix Polynomial Equations: Solution Techniques Garimella Rama Murthy, Associate Professor, IIIT Hyderabad, Gachibowli, HYDERABAD-32, AP, INDIA ABSTRACT In this research
More informationSolving Dense Linear Systems I
Solving Dense Linear Systems I Solving Ax = b is an important numerical method Triangular system: [ l11 l 21 if l 11, l 22 0, ] [ ] [ ] x1 b1 = l 22 x 2 b 2 x 1 = b 1 /l 11 x 2 = (b 2 l 21 x 1 )/l 22 Chih-Jen
More informationComputational Linear Algebra
Computational Linear Algebra PD Dr. rer. nat. habil. Ralf Peter Mundani Computation in Engineering / BGU Scientific Computing in Computer Science / INF Winter Term 2017/18 Part 2: Direct Methods PD Dr.
More informationThe Solution of Linear Systems AX = B
Chapter 2 The Solution of Linear Systems AX = B 21 Upper-triangular Linear Systems We will now develop the back-substitution algorithm, which is useful for solving a linear system of equations that has
More information1 - Systems of Linear Equations
1 - Systems of Linear Equations 1.1 Introduction to Systems of Linear Equations Almost every problem in linear algebra will involve solving a system of equations. ü LINEAR EQUATIONS IN n VARIABLES We are
More information7.6 The Inverse of a Square Matrix
7.6 The Inverse of a Square Matrix Copyright Cengage Learning. All rights reserved. What You Should Learn Verify that two matrices are inverses of each other. Use Gauss-Jordan elimination to find inverses
More informationReview Let A, B, and C be matrices of the same size, and let r and s be scalars. Then
1 Sec 21 Matrix Operations Review Let A, B, and C be matrices of the same size, and let r and s be scalars Then (i) A + B = B + A (iv) r(a + B) = ra + rb (ii) (A + B) + C = A + (B + C) (v) (r + s)a = ra
More informationJACOBI S ITERATION METHOD
ITERATION METHODS These are methods which compute a sequence of progressively accurate iterates to approximate the solution of Ax = b. We need such methods for solving many large linear systems. Sometimes
More informationNeural Networks. Prof. Dr. Rudolf Kruse. Computational Intelligence Group Faculty for Computer Science
Neural Networks Prof. Dr. Rudolf Kruse Computational Intelligence Group Faculty for Computer Science kruse@iws.cs.uni-magdeburg.de Rudolf Kruse Neural Networks 1 Hopfield Networks Rudolf Kruse Neural Networks
More informationMarkov Chains and MCMC
Markov Chains and MCMC Markov chains Let S = {1, 2,..., N} be a finite set consisting of N states. A Markov chain Y 0, Y 1, Y 2,... is a sequence of random variables, with Y t S for all points in time
More informationChapter 7. Tridiagonal linear systems. Solving tridiagonal systems of equations. and subdiagonal. E.g. a 21 a 22 a A =
Chapter 7 Tridiagonal linear systems The solution of linear systems of equations is one of the most important areas of computational mathematics. A complete treatment is impossible here but we will discuss
More informationStochastic processes. MAS275 Probability Modelling. Introduction and Markov chains. Continuous time. Markov property
Chapter 1: and Markov chains Stochastic processes We study stochastic processes, which are families of random variables describing the evolution of a quantity with time. In some situations, we can treat
More informationRadial-Basis Function Networks
Radial-Basis Function etworks A function is radial () if its output depends on (is a nonincreasing function of) the distance of the input from a given stored vector. s represent local receptors, as illustrated
More information10.2 ITERATIVE METHODS FOR SOLVING LINEAR SYSTEMS. The Jacobi Method
54 CHAPTER 10 NUMERICAL METHODS 10. ITERATIVE METHODS FOR SOLVING LINEAR SYSTEMS As a numerical technique, Gaussian elimination is rather unusual because it is direct. That is, a solution is obtained after
More informationNotes on Row Reduction
Notes on Row Reduction Francis J. Narcowich Department of Mathematics Texas A&M University September The Row-Reduction Algorithm The row-reduced form of a matrix contains a great deal of information, both
More informationArtificial Neural Network
Artificial Neural Network Eung Je Woo Department of Biomedical Engineering Impedance Imaging Research Center (IIRC) Kyung Hee University Korea ejwoo@khu.ac.kr Neuron and Neuron Model McCulloch and Pitts
More informationGROUP THEORY PRIMER. New terms: tensor, rank k tensor, Young tableau, Young diagram, hook, hook length, factors over hooks rule
GROUP THEORY PRIMER New terms: tensor, rank k tensor, Young tableau, Young diagram, hook, hook length, factors over hooks rule 1. Tensor methods for su(n) To study some aspects of representations of a
More informationNeural Networks. Mark van Rossum. January 15, School of Informatics, University of Edinburgh 1 / 28
1 / 28 Neural Networks Mark van Rossum School of Informatics, University of Edinburgh January 15, 2018 2 / 28 Goals: Understand how (recurrent) networks behave Find a way to teach networks to do a certain
More informationNumerical Linear Algebra
Chapter 3 Numerical Linear Algebra We review some techniques used to solve Ax = b where A is an n n matrix, and x and b are n 1 vectors (column vectors). We then review eigenvalues and eigenvectors and
More informationSolving Linear Systems of Equations
November 6, 2013 Introduction The type of problems that we have to solve are: Solve the system: A x = B, where a 11 a 1N a 12 a 2N A =.. a 1N a NN x = x 1 x 2. x N B = b 1 b 2. b N To find A 1 (inverse
More informationLinear Systems of n equations for n unknowns
Linear Systems of n equations for n unknowns In many application problems we want to find n unknowns, and we have n linear equations Example: Find x,x,x such that the following three equations hold: x
More informationCSC242: Intro to AI. Lecture 21
CSC242: Intro to AI Lecture 21 Administrivia Project 4 (homeworks 18 & 19) due Mon Apr 16 11:59PM Posters Apr 24 and 26 You need an idea! You need to present it nicely on 2-wide by 4-high landscape pages
More informationDefinition: A binary relation R from a set A to a set B is a subset R A B. Example:
Chapter 9 1 Binary Relations Definition: A binary relation R from a set A to a set B is a subset R A B. Example: Let A = {0,1,2} and B = {a,b} {(0, a), (0, b), (1,a), (2, b)} is a relation from A to B.
More informationMTH 2032 Semester II
MTH 232 Semester II 2-2 Linear Algebra Reference Notes Dr. Tony Yee Department of Mathematics and Information Technology The Hong Kong Institute of Education December 28, 2 ii Contents Table of Contents
More informationChance and Zero Polynomials
Chance and Zero Polynomials by Garimella Ramamurthy Report No: IIIT/TR/2015/-1 Centre for Security, Theory and Algorithms International Institute of Information Technology Hyderabad - 500 032, INDIA August
More informationc c c c c c c c c c a 3x3 matrix C= has a determinant determined by
Linear Algebra Determinants and Eigenvalues Introduction: Many important geometric and algebraic properties of square matrices are associated with a single real number revealed by what s known as the determinant.
More informationAN ITERATION. In part as motivation, we consider an iteration method for solving a system of linear equations which has the form x Ax = b
AN ITERATION In part as motivation, we consider an iteration method for solving a system of linear equations which has the form x Ax = b In this, A is an n n matrix and b R n.systemsof this form arise
More informationMore chapter 3...linear dependence and independence... vectors
More chapter 3...linear dependence and independence... vectors It is important to determine if a set of vectors is linearly dependent or independent Consider a set of vectors A, B, and C. If we can find
More informationIntroduction to Matrices and Linear Systems Ch. 3
Introduction to Matrices and Linear Systems Ch. 3 Doreen De Leon Department of Mathematics, California State University, Fresno June, 5 Basic Matrix Concepts and Operations Section 3.4. Basic Matrix Concepts
More information