Assignment # 4. K-WTA neural networks are relatively simple. N neurons, with all the connection strengths equal to -1 (with no self-connections).

Size: px

Start display at page:

Download "Assignment # 4. K-WTA neural networks are relatively simple. N neurons, with all the connection strengths equal to -1 (with no self-connections)."

Ambrose Garrett
6 years ago
Views:

1 Farrukh Jabeen COM 58 Mathematical Methods In AI Assignment # 4 This problem asks you to analyze a K-WTA network (K winner take all). First read the K-WTA reference article, then answer the questions below. Read the Hopfield reference for a more general introduction to the topic. The power point presentation is also helpful as an introduction. K-WTA neural networks are relatively simple. N neurons, with all the connection strengths equal to - (with no self-connections)..define the connection strength matrix W. Answer: w ij = connection strength between neuron i and neuron j. Connection strengths are symmetric: w ij = w ji. w ij = - if i is not equal to j w ii = 0 (no self-connections) W is an nxn symmetric matrix. In java we can write like that for ( i=0; i < this.m_numberofneurons; i++) { for (int j=0; j < this.m_numberofneurons; j++) { // Setting initial Weights if (i==j) { this.m_weights[i][j] = 0; } else { this.m_weights [i][j] = -; } }

2 2. What are the eigenvalues and eigenvectors of W. (Work out the cases for N=2 and N=3, then try to generalize the result to arbitrary N). Eigenvectors exist only for square matrices. An eigenvector of a square matrix has the defining property that multiplication by this matrix is equivalent to multiplication by a scalar(i.e. the length of the vector changes, its direction does not). If v is an eigenvector of W, Wv= λv Where λ is a scalar called the eigenvalue associated with the eigenvector v. For N=2 W= Computing Eigenvalues and Eigenvectors We can rewrite the condition Wv= v as (W I)v=0 where I is the n n identity matrix the determinant of W I must equal 0. We call p( )=det(w I) the characteristicc polynomial of W. The eigenvalues of W are simply the roots of the characteristic polynomial of W. p( )= det 0-λ λ = λ 2 -=0 = λ 2 = = λ = + Thus, = and 2= are the eigenvalues of W Now we find the eigenvectors corresponding to =. Let (W- I)v=0 (here =) v v= v2 Then from which we obtain the equations v v2 =0 v v2 = 0 If we let v= =t, then v2= t. All eigenvectors corresponding to = are multiples of and thus the eigenspace corresponding to = is given by the span of the eigenspace corresponding to = =.. That is, is a basis of Similarly we can find the eigenvectors corresponding to 2= -

3 All eigenvectors corresponding to 2=- are multiples of and thus the eigenspace corresponding to 2=- is given by the span of. That is eigenspace corresponding to 2=-. s, is a basis of the For N=3 W= Similarly we can find p( )=det(w I) the characteristic polynomial of W. The eigenvalues of W are simply the roots of the characteristicc polynomial of W P( ) 0- λ =det λ λ = -λ λ λ Expand the determinant and we will get the values of λ = -2,, =-2, 2= and λ3= are the eigenvalues of W. Now we find the eigenvectors corresponding to =-2. Let v= v v2 v3 then (W- (-2)I)v=0 = (W +2I)v=0 W + 2I = Now (W+ 2I) v=0 = 0 0 0

4 2v v2- v3=0 -v + 2v2-v3=0 -v v2+ 2v3=0 If we let v3= =t, then v=t and v2=t. All eigenvectors corresponding to =-2 are multiples of and thus the eigenspace corresponding to =-2 is given by the span of eigenspace corresponding to =-2. That is, is a basis of the Similarly we can find the eigenvectors corresponding to 2= All eigenvectors corresponding to 2= are multiples of 0 and thus the eigenspace corresponding to 2= is given by the span of the eigenspace corresponding to 2= =. 0. That is, 0 is a basis of And the eigenvectors corresponding to 3= will be same as above Eigenvalues and eigenvectors can be complex-valued as well as real-valued. The dimension of the eigenspace corresponding to an eigenvalue is less than or equal to the multiplicity of that eigenvalue. The techniques used here are practical for 2 2 and 3 3 matrices. Eigenvalues and eigenvectors of larger matrices are often found using other techniques, such as iterative methods. Let W be an n n matrix. The eigenvalues of W are the roots of the characteristic polynomial p( )=det(w I) For each eigenvalue, we find eigenvectors v= (W I)v=0 The set of alll vectors v satisfying Wv= v is called the eigenspace of W corresponding to. v v2... vn 3. Show that all K-WTA hyperplanes are parallel. by solving the linear system

5 Consider a symmetric network which bounded continuous activation values We will call K winner network if it satisfies the following conditions. It always converges to a final state consisting of K units with maximum activation value and N-k units with minimum activation value. The units at the maximum values are those units that had larger initial states. The permissible exceptions are when there are ties between the winners. The initial state of a unit is defined to be the value of Where is the external input. Here we assume that external inputs are constant with respect to time but can be different for each unit. k-winner corners of the N dimensional hypercube formed by the M and m boundaries. Now let The set of K winner corners for a given K and N. If Then exactly K of are equal to M and the rest are equal to m. So for a given choice of K all the lie on a common N- dimensional hyperplane perpendicular to the vector [,]. Thus, all k-winner hyperplanes are parallel. 4. Show that the k-winner states are stable equilibriums when the external input satisfies (assume m=0 and M=): k- < ext < k. Hint: K-winner states are the corners of the hypercube corresponding to k neurons at maximum activation and the rest at minimum activation. To show that these states are stable equilibriums you must imagine the activation vector as being very close to, but not on, the hypercube corner. Then show that the dynamics of the network will "drive" the activation vector into the corner, as opposed to away from the corner. That is, if activation is near the maximum, then the update will add a positive amount, and if an activation is near the minimum the update will add a negative amount. Here we consider maximum neuron activation = minimum neuron activation = 0 K-winner states are the corners of the hypercube corresponding to k neurons at maximum activation and the rest at minimum activation. To show that these states are stable equilibriums we must imagine the activation vector as being very close to, but not on, the hypercube corner. Then we show that the dynamics of the network will "drive" the activation vector into the corner, as opposed to away from the corner. That is, if activation is near the maximum, then the update will add a positive amount, and if an activation is near the minimum the update will add a negative amount k-winner states are at a stable equilibrium when the neuron activations are no longer changing from one iteration to the next. Neuron activations get updated through the following:

6 () ai(t+) = ai(t) + step * (M - ai(t)) * (ai(t) - m) * neti(t) where 'M' is the maximum activation, 'm' is the minimum activation, step is a constant that determines how fast the system converges to a stable equilibrium, and 'net' is the net input to a neuron. The system enters into stable equilibrium when ai(t+) = ai(t) for all neurons. In our system, this occurs when all neurons have an activation of either 'M' or 'm'. When this occurs, (M - ai(t)) * (ai(t) - m) = 0 and then ai(t+) = ai(t) for all neurons. As the activation vector approaches a hypercube corner, (M - ai(t)) * (ai(t) - m) approaches a minimum (0 in our case), causing the neurons to get closer to either 'm' or 'M' by increasingly smaller increments. However, in order for the neuron's activations to get closer to the hypercube corner, the neti(t) factor must be positive for the neurons approaching 'M' (winners) and negative for the neurons approaching 'm ' (losers). (2a) for k winners: neti(t) > 0 (2b) for N-k losers: neti(t) < 0 neti(t) is calculated through: (3) neti(t) = [Wa(t) + e(t)]i where W is an NxN matrix of connection strengths (wij) populated with 0 where i = j and - everywhere else and e is our external input. In the hypercube corner where k neurons have 'M' activation ( in our case) and N-k neurons have 'm' activation (0 in our case), the following is true: (4a) for k winners: neti(t) = - (k - ) + ei (4b) for N-k losers: neti(t) = - k + ei However, as the system approaches the hypercube corner, the winners aren't quite all at 'M' and the losers aren't quite all at 'm' so, (5a) for k winners: neti(t) < - (k - ) + ei (5b) for N-k losers: neti(t) > - k + ei With our assumptions regarding the sign of neti(t) from (2), this becomes: (6a) for k winners: 0 < - (k - ) + e (6b) for N-k losers: 0 > - k + e Solving for e yields: (7) k - < e < k 5. Show that the dynamics defined by da/dt = net is a gradient descent on the energy function. Gradient descent is a first-order optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or the approximate gradient) of the function at the current point. Suppose in the k winner network if we have number of neurons = 20 maximum neuron activation = minimum neuron activation = 0 number of iterations =500 The following graphs show that energy decreases for each iteration. Here Octave is used to plot the results

7 Gradient descent with respect to an network is defined to be energy function, implemented by calculations. The energy of the A corner seeking part implemented and So interactive activation can be viewed geometrically as a two step process Step : For each unit calculate the negative gradient via. The vector points into particular orthant of N- dimensional space. That orthant is consistent with a particular corner of the N-dimensional hypercube, call it corner(t). From the vector difference between the current activations a(t) and this corner vector: corner(t) a(t).

8 Step2: The multiplying gives the components of the vector that the dynamics will follow That is where From this geometric point of view we can say that that the dynamics follow a vector that points into an orthant that is on the positive side of the gradient descent; thus, for sufficiently small step size, the energy of network decreases at each iteration. References Dr.Wolfe class notes PA4&dq=eigenvalues+neural+network&source=bl&ots=svMyC_UB_N&sig=OCRpobxfriavqpLB5hG_5FdhqU&hl=en&ei=RLZSv3DKoP 8sQPSrbGSBg&sa=X&oi=book_result&ct=result&resnum=5&ved=0CB4Q6AEwBDgU#v=onepage&q=eigenvalues%20neural%20network&f =false

Eigenvalues and Eigenvectors. Prepared by Vince Zaccone For Campus Learning Assistance Services at UCSB

Eigenvalues and Eigenvectors. Prepared by Vince Zaccone For Campus Learning Assistance Services at UCSB Eigenvalues and Eigenvectors Consider the equation A x = λ x, where A is an nxn matrix. We call x (must be non-zero) an eigenvector of A if this equation can be solved for some value of λ. We call λ an