Linear Transforations
Hopfield Network Questions Initial Condition Recurrent Layer p S x W S x S b n(t + ) a(t + ) S x S x D a(t) S x S S x S a(0) p a(t + ) satlins (Wa(t) + b) The network output is repeatedly ultiplied by the weight atrix W. What is the effect of this repeated operation? Will the output converge, go to infinity, oscillate? In this chapter we want to investigate atrix ultiplication, which represents a general linear transforation. 2
Linear Transforations A transforation consists of three parts:. A set of eleents X { }, i called the doain, 2. A set of eleents Y { i }, called the range, and 3. A rule relating each i X to an eleent i Y. A transforation is linear if:. For all, 2 X, ( + 2 ) ( ) + ( 2 ), 2. For all X, a R, (a ) a ( ). 3
Exaple - Rotation Is rotation linear? ( ) (a ) a ( ) a. θ ( ) + 2 ( + 2 ) ( ) 2 2. ( 2 ) 4
Matrix Representation - () Any linear transforation between two finite-diensional vector spaces can be represented by atrix ultiplication. Let {, 2,..., n } be a basis for X, and let {, 2,..., } be a basis for Y. n x i i y i i i i Let :X Y n j x j j i y i i 5
Matrix Representation - (2) Since A is a linear operator, n j x j j i y i i Since the i are a basis for Y, j i a ij u i (The coefficients a ij will ake up the atrix representation of the transforation.) n x j j i a ij i i y i i
Matrix Representation - (3) i i n j a ij x j i y i i n a ij x j y i 0 i j Because the i are independent, n j a ij x j y i This is equivalent to atrix ultiplication. a a 2 a n a 2 a 22 a 2n a a 2 a n x x 2 x n y y 2 y 7
Suary A linear transforation can be represented by atrix ultiplication. To find the atrix which represents the transforation we ust transfor each basis vector for the doain and then expand the result in ters of the basis vectors of the range. j i a ij u i Each of these equations gives us one colun of the atrix. 8
Exaple - () Stand a deck of playing cards on edge so that you are looking at the deck sideways. Draw a vector x on the edge of the deck. Now skew the deck by an angle θ, as shown below, and note the new vector y A(x). What is the atrix of this transforation in ters of the standard basis set? θ ( ) 2 ( ) 9
Exaple - (2) To find the atrix we need to transfor each of the basis vectors. j i a ij u i We will use the standard basis vectors for both the doain and the range. A s j 2 a j s + a 2 j s 2 ( ) a ij s i i 0
Exaple - (3) We begin with s : If we draw a line on the botto card and then skew the deck, the line will not change. A(s ) s A ( s ) s 0s 2 + a i s i 2 a s + a 2 s 2 i This gives us the first colun of the atrix.
Exaple - (4) Next, we skew s 2 : s 2 A(s 2 ) θ tan(θ) A s 2 ( ) tan ( θ ) s + s 2 a i2 s i 2 a 2 s + a 22 s 2 i This gives us the second colun of the atrix. 2
Exaple - (5) The atrix of the transforation is: A tan( θ ) 0 3
Change of Basis Consider the linear transforation :X Y. Let {, 2,..., n } be a basis for X, and let {, 2,..., } be a basis for Y. n x i i y i i i i The atrix representation is: a a 2 a n x y a 2 a 22 a 2n a a 2 x 2 a n x n y 2 y Ax y 4
New Basis Sets Now let s consider different basis sets. Let {, 2,..., n } be a basis for X, and let {, 2,..., } be a basis for Y. n i x' i i i y' i i The new atrix representation is: a' a' 2 a' n x' y' a' 2 a' 22 a' 2n x' 2 a' a' 2 a' n x' n y' 2 y' A'x' y' 5
How are A and A' related? Expand t i in ters of the original basis vectors for X. i n j t ji j t i t i t 2i t ni Expand i in ters of the original basis vectors for Y. i j w ji j w i w i w 2i w i
How are A and A' related? B t t t 2 t n x x' t + x' 2 t 2 + + x' n t n B t x' B w w w 2 w y B w y' Ax y AB t x' B w y' [ B w AB t ]x' y' A'x' y' AB t A' [ B w ] Siilarity Transfor 7
Exaple - () Take the skewing proble described previously, and find the new atrix representation using the basis set {s, s 2 }. 2 2 0.5 + 2 2 0.5 t t 2 B t 0.5 t t 2 B w B t 0.5 (Sae basis for doain and range.) 8
Exaple - (2) AB t A' [ B w ] 2 3 2 3 2 3 3 tanθ 0 0.5 A' ( 2 3) tanθ + ( 2 3) tanθ ( 2 3) tanθ ( 2 3) tanθ + For θ 45 : A' 5 3 2 3 2 3 3 A 0 9
Exaple - (3) Try a test vector: x 0.5 x' 0 y Ax 0.5 0.5 y' A'x' 5 3 2 3 2 3 3 0 5 3 2 3 2 ( ) 2 Check using reciprocal basis vectors: y' B y 0.5.5 2 3 2 3.5 2 3 3 5 3 2 3 20
Eigenvalues and Eigenvectors Let :X X be a linear transforation. Those vectors X, which are not equal to zero, and those scalars λ which satisfy ( ) λ are called eigenvectors and eigenvalues, respectively. ( ) Can you find an eigenvector for this transforation? 2
Coputing the Eigenvalues Az λz [ A λi ]z 0 [ A λi ] 0 Skewing exaple (45 ): A 0 λ 0 λ 0 ( λ ) 2 0 λ λ 2 λ 0 λ z z 0 0 z z 2 0 0 z 0 0 00 00 z 2 0 0 For this transforation there is only one eigenvector. 22
Diagonalization Perfor a change of basis (siilarity transforation) using the eigenvectors as the basis vectors. If the eigenvalues are distinct, the new atrix will be diagonal. B z z 2 z n { z, z 2,, z n } Eigenvectors { λ, λ 2,, λ n } Eigenvalues λ 0 0 [ B AB] 0 λ 2 0 0 0 λ n 23
Exaple A λ λ 0 λ 2 2λ ( λ )( λ 2 ) 0 λ 0 λ 2 2 λ λ z 0 0 λ 0 z z 0 z 2 z z z 2 0 λ z 0 2 2 z z 22 z 2 2 z 22 0 z 2 Diagonal For: A' [ B AB] 2 2 2 2 00 02 24