Dynamics of Ordinary and Recurrent Hopfield Networks: Novel Themes

2017 IEEE 7th International Advance Computing Conference Dynamics of Ordinary and Recurrent Hopfield Networks: Novel Themes Siva Prasad Raju Bairaju, Rama Murthy Garimella, Ayush Jha,Anil Rayala APIIIT RGUKT RKValley sraju728@gmailcom International Institute of Information Technology,Hyderabad rammurthy@iiitacin, anilrayala@studentsiiitacin Indian Institute of Technology, Guwahati ayushjha@iitgernetin Abstract In this research paper several interesting themes related to Hopfield network are explored and concrete results are derived For instance, convergence in partial parallel mode of operation, convergence when synaptic weight matrix, W is row diagonally dominant case Also, structured recurrent Hopfield network with companion matrix as W is studied The dynamics when the activation function is a step function, the state space is asymmetric hypercube are also studied Finally several experimental results are derived Index Terms Structured recurrent Hopfield network, asymmetric hypercube, activation function, companion matrix Dynamics with synaptic matrix as a Row Diagonal Dominant Matrix and Companion matrix are discussed in section IV and V respectively Dynamics in case of step function as an activation function is discussed in section VI and Experimental results with various weight synaptic matrices is discussed in section VII Applications of the proposed Structured Recurrent Hopfield Network are discussed in section VIII Future Work is discussed in section IX The paper concludes in section X II RECURRENT HOPFIELD NEURAL NETWORK I INTRODUCTION Hopfield proposed an artificial neural network(ann) which acts as an associative memory This neural network was subjected to intensive research and several results are reported But certain research issues related to such an ANN remain unresolved For instance, convergence issues related to partial(semi) parallel modes of operation remain unexplored In this research paper we address this problem Also dynamics of Hopfield neural network(hnn) whose synaptic weight matrix is diagonally dominant is explored in this paper RHNN is also a ANN The fundamental feature of a recurrent neural network is that network contains at least one feedback connection, so the activations can flow round in a loop As shown in the fig 1 output of one neuron will act as the input for another neuron Thus, connections in the network can circulate continuously Fig 1 Recurrent Hopfield Network Dynamics of ANN (in the spirit of Hopfield Neural Network) whose synaptic weight matrix is asymmetric is also addressed only recently [2] Such a neural network, called Recurrent Hopfield Neural Network(RHNN) was subjected to experimental investigation In this research paper a structured RHNN whose synaptic weight matrix is a companion matrix is studied Several other issues related to HNN are explored If weight matrix is symmetric then we call it as Ordinary Hopfield Network and when it is asymmetric then we call it as Recurrent Hopfield Network This research paper is designed as follows: In section II, Recurrent Hopfield Neural Network is introduced Convergence in partial parallel mode operation is discussed in section III It is a dynamical system which is non-linear and is represented by a weighted, directed graph The nodes of the graph repre- 978-1-5090-1560-3/17 $3100 2017 IEEE 6 DOI 101109/IACC20179

sent neurons and the edges of the graph represent the weights Each neuron contains a activation (threshold) value Thus, an RHNN can be represented by a thresholding vector T and a synaptic weight matrix W Each neuron assumes a state which lies in {-1,1} The number of neurons represents the order of the network Let the state of the i th neuron at time t be denoted by V i (t) Thus the state of the non-linear dynamical system having N neurons is represented by N 1 vector, V(t) The state updation at the i th node is represented by the following equation, N V i (t +1)=sign[ W ij V j (t) T i ] j=1 where sign is equal to signum function defined as { +1 if t 0 sign(t) = 1 if t<0 Depending on the state updation, the Recurrent hopfield network operation can be in the following modes of operation Serial Mode: If the updation takes place on only one neuron at an instant of time then this type of updation is said to be Serial mode Fully Parallel Mode: If the state updation takes place at all neurons at a time instant, it is called the Fully Parallel Mode Partial(Semi) Parallel Mode: If the state updation takes place simultaneously on only some selected neurons at a time instant, then the network is said to be Partial(Semi) Parallel Mode Remark: Dynamics of Recurrent Hopfield Neural Network(RHNN) is exactly same as that of Ordinary Hopfield Neural Network(OHNN)if and only if the matrix W is symmetric For the cause of integrity, We now include the dynamics of OHNN with W being a symmetric matrix In the state space of (OHNN),a non-linear dynamical system, there are certain well-known states, called the stable states Definition: A state, V (t) is called a stable state if and only if V (t) =sign[w V (t) T ] Thus, if the HNN is in its stable state then the mode of operation cannot affect the HNN Thus, there is no any change in HNN when a stable state is reached III CONVERGENCE IN PARTIAL PARALLEL MODE Theorem 1 If a Hopfield network with symmetric matrix as synaptic matrix is operated in partial(semi) parallel mode, then the network either converges to a stable state or a cycle of length 2 is achieved, after finite number of iterations Proof: The proof is similar to the proof presented in [1] Let R=(M,T) be our original network with n number of nodes/neurons and let us assume that a new network ˆR= ( ˆM, ˆT), with M = [ ] 0 M, T = M 0 [ ] T T The new network R is a hopfield network with 2n number of nodes and also bipartite graph due to our construction The total set of neurons in R can be divided into two disjoint sets, let them be D1 and D2 such that no two neurons in the same set has a non zero weight It is possible because, R is a bipartite graph We will also find that for every neuron 2 D1, there exists a similar neuron 2 D2 which has same edge set There exists a serial mode of operation in R which is equivalent to a partial mode of operation in R Now, we will look into the details let V 0 be the initial state of R and (V 0,V 0 ) be the inital state of R Now, clearly, partial parallel updates in R is equivalent to partial parallel update in D1 and partial parallel update in D2 and proceeding with different possible orders, in the sense that state of R is equal to either state of D1 or D2 depending upon which set, the last evaluation is performed Now, we know there are no internal edge connections in D1 or D2, which implies that, partial parallel update in D1 or D2 is equal to a serial update Hence for every partial parallel update in R, there exist equivalent serial mode updates in R Since we already know that serial mode in a Hopfield network with symmetric synaptic matrix converges to a stable state, that implies, we reach a state state in R Finally, state of R is equal to either state of D1 or D2 depending on last evaluation So, if D1 and D2 are equal, we end up with a stable state for R otherwise we get a cycle of length 2 The Theorem is also empirically evaluated IV DYNAMICS IN CASE OF ROW DIAGONAL DOMINANT MATRIX In this section we study the dynamics of Special case of Recurrent Hopfield Network for fully parallel mode of operation where the synaptic matrix W is a strictly diagonally dominant matrix ie each diagonal element W ii is larger in magnitude than the sum of the magnitudes of all other elements in the i th row W ij < W ii j=1,j i Some of the properties of a diagonal matrix are: 7

It is nonsingular Gaussian elimination can be applied to it without performing any row interchanges Theorem 2 If a Hopfield network with row diagonally dominant matrix as synaptic weight matrix is operated in fully parallel mode, then the network either converges to stable state or a cycle of length 2 Proof: Case 1: All the diagonal elements are positive Here, if the network is operated in parallel update mode, then convergence is observed Let the initial state of the network be V 1 and weight matrix be W Now, due to the structure of our W (row dominant matrix with positive diagonal elements), we can say that, even after a state update, the state of the network will be V 1 which means that network reached a stable state The structure of W ensures that stable state is reached in one step for some initial conditions For other initial conditions, cycle of length 2 is reached Ex: 192540 07480 06127 06173 10326 38180 14724 05795 09766 14248 191375 11928 14340 17267 00501 161111 V 1 = [1 1 1 1] T Here V 1 be the initial state of the network Then, it converges to stable state that is V 1 = [1 1 1 1] T The above reasoning can be generalized to any case because of the structure of our synaptic matrix W Case 2: All the diagonal elements are negative Let the initial state of the network be V 1 and weight matrix be W Now, due to the structure of our W (row dominant matrix with negative diagonal elements), we can say that, even after a state update, the state of the network will be V 1 that means it is a stable state 8 2 3 1 2 10 2 4 1 2 12 5 2 4 1 9 V 1 = [ 1 1 1 1] T Here V 1 be the initial state of the network Then, it will converge to cycle of length 2 (ie, cycles of the state space are of length 2 ) If we start update from state V 1 = [ 1 1 1 1] T then it will go to state V 2 = [1 1 1 1] T and then it will oscillate between two states The above reasoning can be generalized to any case because of the structure of our synaptic matrix W Case 3: Rephrase some diagonal elements are positive and some are negative Let the initial state of the network be V 1 and weight matrix be W Now, due to the structure of our W (row dominant matrix with some negative, some positive diagonal elements), we can say that, even after a state update, the state of the network will be V 1 which means the network reached a stable state 8 2 3 1 2 10 2 4 1 2 12 5 2 4 1 9 V 1 = [1 1 1 1] T Here V 1 be the initial state of the network Then, it converges to cycle of length 2 (ie, cycles of the state space are of length 2) If we start update from state V 1 = [1 1 1 1] T then it will go to state V 2 = [ 1 1 1 1] T and then it will oscillate between two states The above reasoning can be generalized to any case because of the structure of our synaptic matrix W V INTERESTING HOPFIELD NETWORK In [2] and [3], authors proposed novel homogeneous recurrent networks based on modulo neuron and ideas related to linear congruential sequence respectively So, we now propose an interesting Hopfield network with the above motivations Consider a discrete time sequence generated by the following non linear dynamical system x(n + M) =sign({a 1 x(n) +a 2 x(n +1)+ + a n x(n + M 1)}) where a is are real numbers Now we represent such a system by choosing an appropriate state vector Let Y be defined as x(n) x(n +1) Y [n] = x(n + M 1) be the state vector associated with our proposed discrete time non-linear dynamical system Let the initial vector x(0) x(1) Y [0] = x(m 1) 8

lie on the unit hypercube Now, with the above choice of initial vector, the vector-matrix difference equation associated with above equation is Y (n +1)=sign(W Y (n)) for n 0 In the above non linear difference equation, the matrix W is given by 0 1 0 0 0 0 1 0 0 0 0 1 a M a M-1 a M-2 a 1 Thus the state transition matrix of our non linear dynamical system is a companion matrix which is interesting Definition:A state, V (t) is called anti-stable state, if and only if V (t) = sign(wv(t)) By the properties of W, if it has a eigenvalue λ then the corresponding eigenvector will be of the form: 1 λ X = λ (n 1) Theorem 3 Recurrent Hopfield network based on Companion asymmetric matrix can have at most two programmed stable states and two programmed anti-stable states on the hypercube Proof: For a companion matrix based recurrent Hopfield network, we can choose 2 states (that are eigenvectors corresponding to positive eigenvalue, on the hypercube as stable state) For a companion matrix based recurrent Hopfield network, we can choose 2 states (that are eigenvectors corresponding to negative eigenvalue, on the hypercube as anti-stable state) For the eigenvector X to lie on unit hypercube λ must be {+1 or -1} Thus corresponding to eigenvalue +1 the following can be stable states : [+1 + 1 + 1] T,[ 1 1 1] T Similarly corresponding to eigenvalue -1, the following can be anti-stable states: [+1 1 +1 1 +1] T, [ 1 +1 1 +1 1] T From above theorem, and the property of A, the only possible values for λ is 1 or -1 to have a valid state Here matrix A = matrix W, Case 1: Suppose λ = 1, then we will get: A X=λ X Now applying signum function on both sides sign(a X) = sign (λ X)=X that is also how we define a stable state The question here is that if X is a stable state, then what about -X? For this thing consider Z =-X then to get the next state sign(a Z) = sign (A -X) = -sign (A X)=-X which again is the definition of stable state So for eigenvalue λ = 1, we get 2 stable statesthat is X, -X Case 2: Suppose it has -1 as a eigenvalue, with corresponding eigenvector lying on the hypercube(from the above property) Then we get: sign(a X) = sign( λ X) = sign(-x) = -X ie,sign(a X)=-X which is the definition of anti-stable states So for eigenvalue λ = -1 we get anti-stable stateswe chose 1 and -1 because these are the only eigenvalues whose corresponding eigenvector is a possible state Experimental results: 1) Structured recurrent Hopfield network: If network is operating in fully parallel mode and A is asymmetric companion & stochastic matrix then the cycle length of the network is bounded which means that the cycle length is a small number Number of instances L 1 L 2 3 L 5 1000 866 88 46 2) Structured recurrent Hopfield network: If the network is operating in fully parallel mode and A is asymmetric companion & rows sum to 2, 5 (>1) Then the network is Number of instances RowsSum L 32 20000 2 20000 20000 5 20000 20000 7 20000 VI ACTIVATION FUNCTION The activation function converts the activation level of a node into an output signal Various types of activation functions are available in use with ANNs Unit step function is one of the activation functions That is { +1 if t 0 step(t) = 0 if t<0 9

If we take this unit step function as a activation function then the following theorems hold 1) If a Hopfield network with symmetric matrix is operated in fully parallel mode, then the network either converges to a stable state or a cycle of length 2 2) If a Hopfield network with arbitrary symmetric matrix is operated in partial(semi) parallel mode, then the network either converges to a stable state or a cycle of length 2 3) If a Hopfield network with asymmetric matrix with positive elements is operated in fully parallel mode, then the network converge to stable state 4)Let N =(W,T)be the neural network If N is operating in partial parallel mode, W is asymmetric matrix with positive elements then the network will converge to a stable state These are all trivial cases VII RECURRENT HOPFIELD NETWORK BASED ON SPECIAL WEIGHT MATRICES: EXPERIMENTAL RESULTS 1) Asymmetric hypercube: symmetric synaptic weight matrix If the network is operating in fully parallel mode, A is asymmetric hypercube symmetric synaptic weight matrix then the network will converge to a stable state or to a cycle of length 2 2) If we take non-negative diagonal asymmetric matrix and the network s operation is in fully parallel mode then cycle length of the network is bounded which means that the cycle length is a small number Number of instances L 1 2 L 5 1000 993 7 IX FUTURE WORK Significance of Companion matrix based recurrent Hopfield network Other structured synaptic weight matrices: convergence results Practical applications of recurrent Hopfield neural network (RHNNs) X CONCLUSION In this research paper, convergence in partial parallel mode of operation is discussed An interesting structured recurrent Hopfield network(with companion synaptic weight matrix) is proposed Convergence when synaptic weight matrix, W is row diagonally dominant case is discussed The dynamics when the activation function is a step function are discussed Experimental results for different types of recurrent Hopfield networks are shown Some applications are proposed REFERENCES [1] Jehoshua Bruck and Joseph W Goodman, A Generalized Convergence Theorem for Neural Networks, IEEE TRANSACTIONS ON INFORMA- TION THEORY, VOL 34, NO 5, SEPTEMBER 1988 [2] G Rama Murthy, Berkay, Moncef Gabbouji, On the dynamics of a recurrent Hopfield Network,International Joint Conference on Neural Networks(IJCNN-2015),July 2015 [3] G Rama Murthy, Multi-dimensional Neural Networks:Unified Theory,New Age International Publishers,New Delhi,2007 [4] JJ Hopfield, Neural Networks and Physical systems with Emergent Collective Computational Abilities, Proceedings of National Academy of Sciences, USA Vol 79,pp 2554-2558, 1982 [5] ZONG-BEN Xu,Guo-QING Hu AND CHUNG-PING KWONG,Asymmetric Hopfield-type Networks: Theory and Applications, 1996 Elsevier Science LtdVol 9, No 3, pp 483-501, 1996 [6] E Goles, Antisymmetrical neural networks, Discrete ApplMath,vol 13,pp97-100,1986 [7] Hwa-Long Gau, Pei Yuan Wub, Companion matrices: reducibility, numerical ranges and similarity to contractions, ELSEVIER, Linear Algebra and its applications 383(2004) 127-142 [8] Donq-Liang Lee, New Stability Conditions for Hopfield Networks in Partial Simultaneous Update Mode, IEEE TRANSACTIONS ON NEURAL NETWORKS VOL 10, NO4,JULY 1999 VIII APPLICATIONS Neural Network such as Hopfield Network have diverse applications Generally they are use for, associative memory: the network is able to memorize some states, patterns Traditionally, associative memories are defined by associating a single state (ie stable state) with the noise corrupted versions of it In [2], the authors proposed the concept of multi-state associative memories by associating a initial condition with a cycle of states 10