Notes on Linear Algebra

Transcription

1 Notes on Linear Algebra Jay R. Walton Department of Mathematics Texas A&M University October 14, Introduction Linear algebra provides the foundational setting for the study of multivariable mathematics which in turn is the bedrock upon which most modern theories of mathematical physics rest including classical mechanics (rigid body mechanics), continuum mechanics (the mechanics of deformable material bodies), relativistic mechanics, quantum mechanics, etc. At the heart of linear algebra is the notion of a (linear) vector space which is an abstract mathematical structure introduced to make rigorous the classical, intuitive concept of vectors as physical quantities possessing the two attributes of length and direction. In these brief notes, vector spaces and linear transformations are introduced in a three step presentation. First they are studied as algebraic objects and a few important consequences of the concept of linearity are explored. Next the algebraic structure is augmented by introducing a topological structure (via a metric or a norm, for example) providing a convenient framework for extending key concepts from the calculus to the vector space setting permitting a rigorous framework for studying nonlinear, multivariable functions between vector spaces. Finally, an inner product structure for vector spaces is introduced in order to define the geometric notion of angle (and as a special case, the key concept of orthogonality) augmenting the notion of length or distance provided by previously by a norm or a metric. 2 Vector Space The central object in the study of linear algebra is a Vector Space which is defined here as follows. Definition 1 A Vector Space over a field F 1 is a set, V, whose elements are called vectors, endowed with two commutative and associative binary operations, called vector addition and scalar multiplication, subject to the following axioms. Copyright c 2014 by Jay R. Walton. All rights reserved. 1 While in general F can be any field, in these notes it will denote either the real numbers R or the complex numbers C. 1

2 1. If a, b V, the their sum a + b is also in V. 2. Vector addition is commutative and associative, i.e. a + b = b + a and all a, b V and a + (b + c) = (a + b) + c for all a, b, c V. 3. There exists a zero element, 0, in V satisfying 0 + a = a for all a V. 4. Every element a V possesses a (unique) additive inverse, a, satisfying a + a = If a V and α F, then αa V. 6. For all a V, 1 a = a. 7. If a, b V and α, β F, then α(a+b) = αa+αb, (α+β)a = αa+βa, α(βa) = (αβ)a. 2 Remark 1 It follows readily from these axioms that for any a V, 0a = 0 and ( 1)a = a. (Why?) Remark 2 The above axioms are sufficient to conclude that for any collection of vectors a 1,..., a N V and scalars α 1,..., α N F, the Linear Combination α 1 a α N a N is an unambiguously defined vector in V. (Why?) Example 1 The prototype examples of vector spaces over the real numbers R and the complex numbers C are the Euclidean spaces R N and C N of N-tuples of real or complex numbers. Specifically, the elements a R N are defined by a 1 a :=. where a 1,..., a N R. Vector addition and scalar multiplication are defined componentwise, that is a 1 b 1 αa 1 + βb 1 αa + βb = α. + β. =.. a N b N αa N + βb N Analogous relations hold for the complex case. Example 2 The vector space R is defined as the set of all (infinite) sequences, a, of real numbers, that is, a R means ) a = a N ( a1 with vector addition and scalar multiplication done componentwise. 2 The symbol + has been used to denote both the addition of vectors and the addition of scalars. This should not generate any confusion since the meaning of + should be clear from the context in which it is used.. 2

3 Definition 2 A subset U of a vector space V is called a Subspace provided it is closed with respect to vector addition and scalar multiplication (and hence is also a vector space). Example 3 One shows easily that R N is a subspace of R M whenever N M. (Why?) Example 4 The vector space R f is the set all of infinite sequences of real numbers with only finitely many non-zero components. It follows readily that R f is a subspace of R. (Why?) Definition 3 Let A be a subset of a vector space V. Then the Span of A, denoted span(a), is defined to be the set of all (finite) linear combinations formed from the vectors in A. A subset A V is called a Spanning Set for V if span(a) = V. A spanning set A V is called a Minimal Spanning Set if no proper subset of A is a spanning set for V. Remark 3 One readily shows that span(a) is a subspace of V. (How?) 2.1 Linear Independence and Dependence The most important concepts associated with vector spaces are Linear Independence and Linear Dependence. Definition 4 A set of vectors A V is said to be Linearly Independent if given any vectors a 1,..., a N A, then the only way to write the zero vector as a linear combination of these vectors is if all coefficients are zero, that is, α 1 a α N a n = 0 if and only if α 1 =... = α N = 0. Remark 4 If a set of vectors A is linearly independent and B A, then B is also linearly independent. (Why?) Remark 5 A minimal spanning set is easily seen to be linearly independent. Indeed, suppose A is a minimal spanning set and suppose it is not linearly independent. Then there exists a subset {a 1,..., a N } of A such that a N can be written as a linear combination of a 1,..., a N 1. It follows that A {a N } (set difference) is a proper subset of A that spans V contradicting the assumption that A is a minimal spanning set. Definition 5 If a 0, then span({a}) is called a Line in V. If {a, b} is linearly independent, then span({a, b}) is called a Plane in V. Definition 6 A set of vectors A V is said to be Linearly Dependent if there exists a set of vectors a 1,..., a N V and scalars α 1,..., α N F, not all zero, with α 1 a α N a N = 0. 3

4 Remark 6 A set of vectors that contains the zero vector is linearly dependent. (Why?) Remark 7 If the set of vectors A is linearly dependent and A B, then B is also linearly dependent. (Why?) Remark 8 If A = {a 1,..., a N } V is linearly dependent, then one of the a j can be written as a non-trivial (not all scalar coefficients zero) linear combination of the others, that is, one of the a j is in the span of the others. 2.2 Basis The notion of Basis for a vector space is intimately connected with spanning sets of minimal cardinality. Here attention is restricted to vector spaces with spanning sets containing only finitely many vectors. Definition 7 A vector space V is called Finite Dimensional provided it has a spanning set with finitely many vectors. Theorem 1 Suppose V is finite dimensional with minimal spanning set A = {a 1,..., a N }. Let B be another minimal spanning set for V. Then #(B) = N, where #(B) denotes the cardinality of the set B. Proof: It suffices to assume that #(B) N. (Why?) Then choose B := {b 1,..., b N } B. Clearly, B is linearly independent. (Why?) Since A is spanning, there exist scalars α ij such that b i = N j=1 α ija j. Then the matrix L := [α ij ] is invertible. (Why?) Suppose L 1 = [β ij ]. Then a i = N j=1 β ijb j, that is, the set {a 1,..., a N } is in the span of B. But since A spans V, it follows that B must also span V. But since B was assumed to be a minimal spanning set, it must be that B = B. (Why?) Hence, #(B) = N. 3 This theorem allows one to define unambiguously the notion of dimension for a finite dimensional vector space. Definition 8 Let V be a finite dimensional vector space. Then its Dimension, denoted dim(v), is defined to be the cardinality of any minimal spanning set. Definition 9 A Basis for a finite dimensional vector space is defined to be any minimal spanning set. It follows that given a basis B = {b 1,..., b N } for a finite dimensional vector space V, every vector a V can be written uniquely as a linear combination of the base vectors, that is N a = α j b j (1) j=1 with the coefficients α 1,..., α N being uniquely defined. These coefficients are called the coordinates or components of a with respect to the basis B. It is often convenient collect 3 The symbol is used in these notes to indicate the end of a proof. 4

5 these components into an N-tuple denoted α 1 [a] B :=. Notational Convention: It is useful to introduce simplifying notation for sums of the sort (1). Specifically, in the Einstein Summation Convention, expressions in which an index is repeated are to be interpreted as being summed over the range of the index. Thus (1) reduces to N a = α j b j = α j b j. j=1 With the Einstein summation convention in effect, if one wants to write an expression like α j b j for a particular, but unspecified, value of j, then one must write α N. α j b j (no sum) Example 5 In the Euclidean vector space R N, it is customary to define the Natural Basis N = {e 1,..., e N } to be e 1 := 0., e 2 := 1.,..., e N := Finding components of vectors in R N with respect to the natural basis is then immediate. that is, a 1 a =. a N R N = [a] N = a = a j e j. a 1. In general, finding the components of a vector with respect to a prescribed basis usually involves solving a linear system of equations. For example, suppose a basis B = {b 1,..., b N } in R N is given as Then to find the components b 1 := b 11. b N1,..., b N := α 1 [a] B =. 5 α N b 1N. b NN a N.

6 of a vector a 1 a :=. with respect to B, one solves the linear system of equations a i = b ij α j, i = 1,..., N which in expanded form is 2.3 Change of Basis a N a 1 = b 11 α b 1N α N a N = b N1 α b NN α N. Let B = {b 1,..., b N } and B = { b 1,..., b N } be two bases for the vector space V. The question considered here is how to change components of vectors with respect to the basis B into components with respect to the basis B? To answer the question, one needs to specify some relation between the two sets of base vectors. Suppose, for example, one can write the base vectors b j as linear combinations of the base vectors b j, that is, one knows the components of each N-tuple [ b j ] B, j = 1,..., N. To be more specific, suppose b j = t ij b i, for j = 1,..., N. (2) Form the matrix T := [t ij ] called the Transition Matrix between the bases B and B. Then T is invertible. (Why?) Let its inverse have components T 1 = [s ij ]. By definition of matrix inverse, one sees that s ik t kj = δ ij where the Kronecker symbol δ ij is defined as { 1 when i = j, δ ij := 0 when i j. Suppose now that a vector a has components. α 1 [a] B =. Then one shows readily (How?) that the components of a with respect to the basis B are given by α 1 [a] B =. α N with α N α i = s ij α j.. In matrix notation this change of basis formula becomes [a] B = T 1 [a] B. (3) 6

7 3 Linear Transformations Linear transformations are at the heart of the subject of linear algebra. Important in their own right, they also are the principal tool for understanding nonlinear functions between subsets of vector spaces. They will be studied here first from an algebraic perspective. Subsequently, they will be studied geometrically. Definition 10 A linear transformation, L : is a function satisfying V U, between the vector spaces V and U L(αa + βb) = αla + βlb, for every α, β F and a, b V. 4 (4) Remark 9: The symbol + on the left hand side of (4) denotes vector addition in V whereas on right hand side of (4) it denotes vector addition in U. Remark 10: From the definition (4) it follows immediately that once a linear transformation has been specified on a basis for the domain space, its values are determined on all of the domain space through linearity. Two important subspaces associated with a linear transformation are its Range and Nullspace. Consider a linear transformation L : V U. (5) Definition 11 The Range of a linear transformation (5) is the subset of U, denoted Ran(L), defined by Ran(L) := {u U there exists v V such that u = Lv}. Definition 12 The Nullspace of a linear transformation (5) is the subset of V, denoted Nul(L), defined by Nul(L) := {v Lv = 0}. Remark 9 It is an easy exercise to show that Ran(L) is a subspace of U and Nul(L) is a subspace of V. (How?) Example 6 Let A be an M N matrix A := a 11. a 1N a M1 a MN. = (a 1 a N ) where a 1,..., a N denote the column vectors of A, that is, A is viewed as consisting of N- column vectors from R M. 4 In analysis it is customary to make a notational distinction between the name of a function and the values the function takes. Thus, for example, one might denote a function by f and values the function takes by f(x). For a linear transformation, L, it is customary to denote its values by La. This notation is suggestive of matrix multiplication. 7

8 Definition 13 The Column Rank of A is defined to be the dimension of the subspace of R M spanned by its column vectors. One can define a linear transformation A : R N R M by Av := Av (matrix multiplication). (6) It is readily shown that the range of A is the span of the column vectors of A, and the nullspace of A is the set of N-tuples satisfying Ran(A) = span{a 1,..., a N } v 1 v =. v N R N v 1 a v N a N = 0. It follows that dim(ran(a)) equals the number of linearly independent column vectors of A. One can also view the matrix A as consisting of M-row vectors from R N and define its row rank by Definition 14 The Row Rank of A is defined to be the dimension of the subspace of R N spanned by its row vectors. A fundamental result about matrices is Theorem 2 Let A be an M N matrix. Then Proof: Postponed. A closely related result is Row Rank of A = Column Rank of A. (7) Theorem 3 Let A be an M N matrix and A the linear transformation (6). Then dim(ran(a)) + dim(nul(a)) = dim(r N ) = N. (8) The result (8) is a special case of the more general Theorem 4 Let L : V U be a linear transformation. Then dim(ran(l)) + dim(nul(l)) = dim(v). (9) The proof of this last theorem is facilitated by introducing the notion of Direct Sum Decomposition of a vector space V. 8

9 Definition 15 Let V be a finite dimensional vector space. A Direct Sum Decomposition of V, denoted V = U W, consists of a pair of subspaces, U, W of V with the property that every vector v V can be written uniquely as a sum v = u + w with u U and w W. Remark 10 It follows easily from the definition of direct sum decomposition that V = U W = dim(v) = dim(u) + dim(w). (Why?) Example 7 Two non-colinear lines through the origin form a direct sum decomposition of R 2 (Why?), and a plane and non-coincident line through the origin form a direct sum decomposition of R 3 (Why?). Proof of Theorem 4: Let {f 1,..., f n } be a basis for Nul(L) where n = dim(nul(l)). Fill out this set to form a basis for V, {g 1,..., g N n, f 1,..., f n } (How?). Define the subspace Ṽ := span({g 1,..., g N n }). Then Ṽ and Nul(L) form a direct sum decomposition of V (Why?), that is V = Ṽ Nul(L). Moreover, one can easily show that L restricted to Ṽ is a one-to-one transformation onto Ran(L) (Why?). It follows that {Lg 1,..., Lg N n } forms a basis for Ran(L) (Why?) and hence that dim(ran(l)) = dim(ṽ) = N n (Why?) from which one concludes (9). 3.1 Matrices and Linear Transformations Let L : V U be a linear transformation and let B := {b 1,..., b N } be a basis for V and C := {c 1,..., c M } a basis for U. Then the action of the linear transformation L can be carried out via matrix multiplication of coordinates in the following manner. Given v V, define u := Lv. Then the Matrix of L with respect to the bases B and C is defined to be the unique matrix L = [L] B,C satisfying or in more expanded form [u] C = L[v] B (matrix multiplication) [Lv] C = [L] B,C [v] B. (10) To see how to construct [L] B,C, first construct the N column vectors (M-tuples) l j := [Lb j ] C, that is the coordinates with respect to the basis C in U of the image vectors Lb j of the base vectors in B. Then, one easily shows that defining [L] B,C := [l 1... l N ] (matrix whose columns are l 1,..., l N ) (11) gives the unique M N matrix satisfying (10). Indeed, if v V has components [v] B = [v j ], that is v = v j b j, then [Lv] C = [v j Lb j ] C = [l 1... l N ][v] B as required to prove (11). Remark 11 If U = V and B = C, then the matrix of L with respect to B is denoted [L] B and is constructed by using the B-coordinates of the vectors Lb j, [Lb j ] B, as its columns. Example 8 The identity transformation I : V V (Iv = v for all v V) has where δ ij is the Kronecker symbol. (Why?) [I] B = [δ ij ] 9

10 3.2 Change of Basis Suppose L : V V and B = {b 1,..., b N } and B = { b 1,..., b N } are two bases for V. The question of interest here is: How are the matrices [L] B and [L] B related? The question is easily answered by using the change of basis formula (3). From the definition of [L] B and [L] B one has [Lv] B = [L] B [v] B and [Lv] B = [L] B[v] B. It follows from (3) that from which one concludes Remark 12 [Lv] B = T 1 [Lv] B = T 1 [L] B [v] B = T 1 [L] B T [v] B Square matrices A and B satisfying [L] B = T 1 [L] B T. (12) A = T 1 BT for some (non-singular) matrix T are said to be Similar. Thus, matrices related to a given linear transformation through a change of basis are similar and conversely similar matrices correspond the same linear transformation under a change of basis. 3.3 Trace and Determinant of a Linear Transformation Consider linear transformations on a given finite dimensional vector space V, i.e. L : V. Then its Trace is defined by Definition 16 The trace of L, denoted tr(l), is defined by tr(l) := tr([l] B ) V for any matrix representation of L. To see that this notion of trace of a linear transformation is well (unambiguously) defined, let [L] B and [L] B denote two different matrix representations of L. Then they must be similar matrices, i.e. there exists a non-singular matrix T such that [L] B = T 1 [L] B T. But then from the elementary property of the trace of a square matrix, tr(ab) = tr(ba) for any two square matrices A, B (Why true?), one has tr([l] B) = tr(t 1 [L] B T ) = tr(t T 1 [L] B ) = tr([l] B ). In similar fashion, one can define the Determinant of a linear transformation on V. Definition 17 The determinant of a linear transformation L on a finite dimensional vector space V is defined by det(l) := det([l] B ) where [L] B is any matrix representation of L. That this notion of the determinant of a linear transformation is well defined can be seen from an argument similar to that above for the trace. The relevant property of the determinant of square matrices needed in the argument is det(ab) = det(a) det(b) (Why true?). 10

11 4 Metric, Norm and Inner Product In this section, the algebraic structure for vector spaces presented above is augmented through the inclusion of geometric structure. Perhaps the most basic geometric notion is that of distance or length. A convenient way to introduce a notion of distance into a vector space is through use of a Metric. Definition 18 A Metric on a vector space V is a real valued function of two variables d(, ) : V R satisfying the following axioms. For any a, b, c V, d(a, b) 0, with d(a, b) = 0 a = b (P ositivity) d(a, b) = d(b, a) (Symmetry) (13) d(a, b) d(a, c) + d(c, b). (T riangle Inequality) Remark 13 Given a metric on a vector space, one can define the important concept of a metric ball. Specifically, an (open) metric ball, B(c, r), centered at c and of radius r is defined to be the set B(c, r) := {v V d(c, v) < r}. One can also define a notion of metric convergent sequence as follows. {a j, j = 1, 2,...} V is said to be metric convergent to a V provided lim d(a j, a) = 0. j A sequence An equivalent definition in terms of metric balls states that for all ɛ > 0, there exists N > 0 such that a j B(a, ɛ) for all j > N. An important class of metrics on a vector space comes from the notion of a Norm. Definition 19 A Norm on a vector space is a non-negative, real valued function : V R satisfying the following axioms. For all a, b V and α F, a 0, with a = 0 a = 0 (Positivity) αa = α a (Homogeneity) (14) a + b a + b. (Triangle Inequality) Remark 14 A norm has an associated metric defined by d(a, b) := a b. An important class of norms on a vector space comes from the notion of Inner Product. Here a distinction must be made between real and complex vector spaces. 11

12 Definition 20 An inner product on a real vector space is a positive definite, symmetric bilinear form, : V V R satisfying the following axioms. For all a, b, c V and α R, a, a 0 with a, a = 0 a = 0 (Positivity) a, b = b, a (Symmetry) αa, b = α a, b (Homogeneity) (15) a + c, b = a, b + c, b. (Additivity) Definition 21 An inner product on a complex vector space is a positive definite, conjugatesymmetric form, : V V C satisfying the following axioms. For all a, b, c V and α C, a, a 0 with a, a = 0 a = 0 (Positivity) a, b = b, a (Conjugate-Symmetry) (16) αa, b = α a, b (Homogeneity) a + c, b = a, b + c, b (Additivity) where in (16), b, a denotes the complex conjugate of b, a. It is evident from the definition that a complex inner product is linear in the first variable but conjugate linear in the second. In particular, for all a, b V and α C αa, b = α a, b whereas a, αb = α a, b. Remark 15 Given an inner product on either a real or complex vector space, one can define an associated norm by a := a, a. (Why does this define a norm?) Remark 16 In R N, the usual inner product, defined by a, b := a 1 b a N b N = a j b j where a =. a 1 a N, b = b 1. is often denoted by a b, the so-called dot product. In C N, the usual inner product is given by a, b = a b = a 1 b a N b N where now a j b j C and the over-bar denotes complex conjugation. Example 9 Let A be a positive definite, symmetric, real, N N matrix. 5 Then A defines an inner product on R N through a, b A := a (Ab). 5 Recall that symmetric means that A T = A, where A T denotes the transpose matrix, and positive definite means a (Aa) > 0 for all a R N. 12 b N,

13 Example 10 An important class of norms on R N and C N are the p-norms defined for p 1 by ( N ) 1/p a p := a j p. (17) j=1 One must show that (17) satisfies the axioms (14) for a norm. The positivity and symmetry axioms from (14) are straight forward to show, but the triangle inequality is not. A proof that (17) satisfies the triangle inequality in (14) can be found in the supplemental Notes on Inequalities. The special case p = 2 is the famous Minkowski Inequality for any a, b R N. a + b 2 a 2 + b 2 Example 11 The -norm is defined by a := max{ a 1,..., a N }. (18) The task of showing that (18) satisfies the axioms (14) for a norm is left as an exercise. An important observation about the p-norms is that lim p a p = a whose proof is also left as an exercise. That the -norm is the limit of the p-norms as p is what motivates the definition (18). Remark 17 The 2-norm is very special among the class of p-norms being the only one coming from an inner product. The 2-norm is also called the Euclidean Norm since it comes from the usual inner product on the Euclidean space R N. Remark 18 In a real vector space (V,, ), one can define a generalized notion of angle between vectors. Specifically, given a, b V, one defines the angle, θ, between a and b to be the unique number 0 < θ < π satisfying cos θ = a, b a b. As a special case, two vectors a, b V are said to be Orthogonal (in symbols a b) if a, b = 0. (19) Remark 19 Analytically or topologically all norms on a finite dimensional vector space are equivalent in the sense that if a sequence is convergent with respect to one norm, it is convergent with respect to all norms on the space. This observation follows directly from the following important 13

14 Theorem 5 Let V be a finite dimensional vector space and let I and II be two norms on V. Then there exist positive constants m, M such that for all v V m v I v II M v I. Proof: The proof of this result is left as an exercise. While all norms on a finite dimensional vector space might be topologically equivalent, they are not geometrically equivalent. In particular, the shape of the (metric) unit ball can vary greatly for different norms. That fact is illustrated in the following Exercise 1 Sketch the unit ball in R 2 for the p-norms, 1 p. 4.1 Orthonormal Basis In a finite dimensional inner product space (V,, ), an important class of bases consists of the Orthonormal Bases defined as follows. Definition 22 In an inner product space (V,, ), a basis B = {b 1,..., b N } is called Orthonormal provided b i, b j = δ ij for i, j = 1,..., N. (20) Thus, an orthonormal basis consists of pairwise orthogonal unit vectors. Finding coordinates of arbitrary vectors with respect to such a basis is as convenient as it is for the natural basis in F N. Indeed, what makes the natural basis so useful is that it is an orthonormal basis with respect to the usual inner product on R N. Remark 20 Suppose B = {b 1,..., b N } is an orthonormal basis for (V,, ) and a V. Then one easily deduces that a = a 1 b a N b N = a j = a, b j as can be seen by taking the inner product of the first equation with each base vector b j. Moreover, one has a, c = [a] B [c] B (21) where the right hand side is the usual dot-product of N-tuples in R N [a] B [c] B = a 1 c a N c N. Thus, computing the inner product of vectors using components with respect to an orthonormal basis uses the familiar formula for the dot-product on R N with respect to the natural basis. In similar fashion one can derive a simple formula for the components of the matrix representation of a linear transformation on an inner product space with respect to an orthonormal basis. Specifically, suppose L : V V and B = {b 1,..., b N } is an orthonormal basis. Then it is straightforward to show that L := [L] B = [l ij ] with l ij = b i, Lb j. (22) 14

15 One way to derive (22) is to first recall the formula l ij = e i (Le j ) (23) for the components of an N N matrix, L := [l ij ], where {e 1,..., e N } is the natural basis for R N. It follows from (21) that for all a, c V a, Lc = [a] B ([Lc] B ) = [a] B ([L] B [c] B ). (24) Now letting a = b i, c = b j and noting that [b j ] B = e j for all j = 1,..., N, one deduces (22) from (23,24) General Bases and the Metric Tensor More generally, one would like to be able to conveniently compute the inner product of two vectors using coordinates with respect to some general (not necessarily orthonormal) basis. Suppose B := {f 1,..., f N } is a basis for (V,, ) and let a, b V with coordinates [a] B = [a j ], [b] B = [b j ], respectively. Then one has N a, b = a j f j, j=1 N N b k f k = a j b k f j, f k = [a] B (F [b] B ) k=1 j,k=1 where the matrix F := [ f i, f j ] = [f ij ] (25) is called the Metric Tensor associated with the basis B. It follows that the familiar formula (21) for the dot-product in R N holds in general if and only if the basis B is orthonormal. The matrix F is called a metric tensor because it is needed to compute lengths and angles using coordinates with respect to the basis B. In particular, if v V has coordinates [v] B = [v j ], then v (the length of v) is given by v 2 = v, v = f ij v i v j The Gram-Schmidt Orthogonalization Procedure Given any basis B = {f 1,..., f N } for an inner product space (V,, ), there is a simple algorithm, called the Gram-Schmidt Orthogonalization Procedure, for constructing from B an orthonormal basis B = {e 1,..., e N } satisfying span{f 1,..., f K } = span{e 1,..., e K } for each 1 K N. (26) The algorithm proceeds inductively as follows. Step 1: Define e 1 := f 1 / f 1, where denotes the norm associated with the inner product,. Step 2: Subtract from f 2 its component in the direction of e 1 and the normalize. More specifically, define g 2 := f 2 f 2, e 1 e 1 15

16 and then define e 2 := g 2 g 2. It is easily seen that {e 1, e 2 } is an orthonormal pair satisfying (26) for K = 1, 2. Inductive Step: Suppose orthonormal vectors {e 1,..., e J } have been constructed satisfying (26) for 1 K J with J < N. Define and then define g J+1 := f J+1 f J+1, e 1 e 1... f J+1, e J e J e J+1 := g J+1 g J+1. One easily sees that {e 1,..., e J+1 } is an orthonormal set satisfying (26) for 1 K J + 1 thereby completing the inductive step. 4.2 Reciprocal Basis Let (V,, ) be a finite dimensional inner product space with general basis B = {f 1,..., f N }. Then B has an associated basis, called its Reciprocal or Dual Basis defined through Definition 23 Given a basis B = {f 1,..., f N } on the finite dimensional inner product space (V,, ), there is a unique Reciprocal Basis B := {f 1,..., f N } satisfying f i, f j = δ ij for i, j = 1,..., N. (27) Thus, every vector v V has the two representations v = = N v i f i (Contravariant Expansion) (28) i=1 N v i f i. (Covariant Expansion) (29) i=1 The N-tuples [v] B = [v i ] and [v] B = [v i ] are called the Contravariant Coordinates and Covariant Coordinates of v, respectively. It follows immediately from (27,28,29) that the contravariant and covariant coordinates of v can be conveniently computed via the formulae v i = v, f i and v i = v, f i. (30) Exercise 2 Let B = {f 1, f 2 } be any basis for the Cartesian plane R 2. Show how the reciprocal basis B = {f 1, f 2 } can be constructed graphically from B. Exercise 3 Let B = {f 1, f 2, f 3 } be any basis for the Cartesian space R 3. Show that the reciprocal basis B = {f 1, f 2, f 3 } is given by f 1 = f 2 f 3 f 1 (f 2 f 3 ), f 2 = f 1 f 3 f 2 (f 1 f 3 ), f 3 = f 1 f 2 f 3 (f 1 f 2 ) where a b denotes the vector cross product of a and b. 16

17 5 Linear Transformations Revisited The theory of linear transformations between finite dimensional vector spaces is revisited in this section. After some general considerations, several important special classes of linear transformations will be studied from both an algebraic and a geometric perspective. The first notion introduced is the vector space Lin(V, U) of linear transformations between two vector spaces. 5.1 The Vector Space Lin(V, U) Definition 24 Given two vector spaces, V, U, Lin(V, U) denotes the collection of all linear transformations between V and U. It is a vector space under the operations of addition and scalar multiplication of linear transformations defined as follows. Given two linear transformations L 1, L 2 : V U their sum, (L 1 + L 2 ), is defined to be the linear transformation satisfying (L 1 + L 2 )v := L 1 v + L 2 v for all v V. Moreover, scalar multiplication of a linear transformation is defined by (αl)v := αlv for all α F and v V. For the special case V = U, one denotes the vector space of linear transformations on V by Lin(V) Adjoint Transformation Let L Lin(V, U) be a linear transformation between two inner product spaces (V,, ) and (U,, ). The Adjoint Transformation, L, is defined to be the unique transformation in Lin(U, V) satisfying Lv, u = v, L u for all v V and u U. (31) Remark 21 The right hand side of (31) is the inner product on V whereas the left hand side is the inner product on U. Remark 22 This definition of adjoint makes sense due to the Theorem 6 (Riesz Representation Theorem) Suppose l : V F is a linear, scalar valued transformation on the inner product space (V,, ). Then there exists a unique vector a V such that lv = v, a for all v V. Note that for fixed u U, lv := Lv, u is a scalar valued linear transformation on V. By the Riesz Representation Theorem, there exists a unique vector a V such that lv = v, a for all v V. One now defines L u := a V. 17

18 Example 12 Suppose V = R N, U = R M and L M M,N (R), the space of all M N (real) matrices. Then L = L T, the transpose of L. (Why?) On the other hand, if V = C N, U = C M and L M M,N (C), the space of all M N (complex) matrices. Then L = L T, the conjugate transpose of L. (Why?) Remark 23 Consider the special case V = U (over F = R) and let B := {f 1,..., f N } be any basis for V. Then one can show that for any A Lin(V) where the matrix F is given by with The proof of (32) is left as an exercise. [A ] B = F 1 [A] T BF (32) F := [f ij ] f ij := f i, f j Two Natural Norms for Lin(V, U) One can define a natural inner product on Lin(V) by L 1 L 2 := tr(l 1L 2 ) (33) where L 1 denotes the adjoint of L 1. (Why does this satisfy the axioms (15) or (16)?) Remark 24 In the case V = R N with L 1 and L 2 given by N N- matrices L 1 = [l (1) ij ], L (2) 2 = [l (2) ], respectively, then one readily shows (How?) that ij L 1 L 2 = N i,j=1 l (1) ij l(2) ij. One can readily show that (33) also defines an inner product on Lin(V, U) where V and U are any two finite dimensional vector spaces. (How?) The norm associated to this inner product, called the trace norm, is then defined by L := L L = tr(l L). (34) Another natural norm on Lin(V, U), where (V, ) and (U, ) are normed vector spaces, called the operator norm, is defined by L := sup Lv. (35) v =1 Question: What is the geometric interpretation of (35)? An important relationship between the trace norm and operator norm is given by 18

19 Theorem 7 Let (V,, ) and (U,, ) be two finite dimensional, inner product spaces. Then for all L Lin(V, U), L L N L. (36) Proof: The proof uses ideas to be introduced in the up coming study of spectral theory. To see how spectral theory plays a role in proving (36), first notice that L = sup Lv = sup Lv, Lv = sup v, L Lv. v =1 v =1 v =1 Next observe that L L is a self-adjoint, positive linear transformation in Lin(V). As part of spectral theory for self-adjoint operators on a finite dimensional inner product space, it is shown that sup v, L Lv = v =1 max λ j 2 j=1,...,n where { λ j 2, j = 1,..., N} are the (positive) eigenvalues of L L. Thus, one has L = max λ j. (37) j=1,...,n On the other hand, spectral theory will show that for the trace norm, one has L 2 = tr(l L) = N λ j 2. (38) j=1 Appealing to the obvious inequalities max λ j 2 j=1,...,n one readily deduces (36) from (37,38) Elementary Tensor Product N j=1 λ j 2 N max j=1,...,n λ j 2, Among the most useful linear transformations is the class of Elementary Tensor Products which can be used as the basic building blocks of all other linear transformations. First consider the case of Lin(V) where (V,, ) is an inner product space. Definition 25 Let a, b V. Then the elementary tensor product of a and b, denoted a b, is defined to be the linear transformation on V satisfying (a b) v := v, b a for all v V. (39) Remark 25 It follows immediately from the definition that Ran(a b) = span(a) and Nul(a b) = b, where b denote the orthogonal complement of b, i.e. the (N 1)- dimensional subspace of all vectors orthogonal to b (where dim(v) = N). The elementary tensor products are thus the rank one linear transformations in Lin(V). 19

20 Remark 26 The following identities can also be derived easily from the definition of elementary tensor product. Given any a, b, c, d V and A Lin(V) (a b) = b a (a b) (c d) = a, c b, d A (a b) = a, Ab (a b) (c d) = b, c a d A (a b) = (Aa) b (a b) A = a (A b) tr(a b) = a, b. Example 13 When V = R N and a = (a i ), b = (b i ), then the matrix of a b with respect to the natural basis on R N is [a b] N = [a i b j ]. (Why?). (40) Remark 27 The result (40) has a natural generalization to a general finite dimensional inner product space (V,, ). Let B := {e 1,..., e N } be an orthonormal basis for V and a, b V. Then [a b] B = [a i b j ] (41) where [a] B = [a i ] and [b] B = [b i ]. Hence, the formula (40) generalizes to the matrix representation of an elementary tensor product with respect to any orthonormal basis for V. More generally, if B = {f 1,..., f N } is any (not necessarily orthonormal) basis for V, then [a b] B = [a i b j ]F (matrix product) where F is the metric tensor associated with the basis B defined by (25). Verification of this last formula is left as an exercise. The notion of elementary tensor product is readily generalized to the setting of linear transformations between two inner product spaces (V,, ) and (U,, ) as follows. Definition 26 Let a U and b V. Then the elementary tensor product a b is defined to be the linear transformation in Lin(V, U) satisfying (a b) v := v, b a for all v V A Natural Orthonormal Basis for Lin(V, U) Using the elementary tensor product, one can construct a natural orthonormal basis for Lin(V, U). First consider the special case of Lin(V). Suppose (V,, ) is an inner product space with orthonormal basis B := {e 1,..., e N }. Then one easily shows that the set B := {e i e j, i, j = 1,..., N} provides an orthonormal basis for Lin(V) equipped with the inner product (33). If follows immediately that dim(lin(v)) = N 2. 20

21 Remark 28 If L Lin(V), then L = l ij e i e j with l ij = L, e i e j = e i, Le j. (Why?) (42) More generally, suppose B U = {b 1,..., b M } is an orthonormal basis for U and B V = {d 1,..., d N } is an orthonormal basis for V. Then a natural orthonormal basis for Lin(V, U) (with respect to the trace inner product on Lin(V, U)) is {b i d j i = 1,..., M, j = 1,..., N}. Indeed, that these tensor products are pairwise orthogonal and have trace norm one follows from b i d j, b k d l = tr((b i d j ) b k d l ) The generalization of (42) for L Lin(V, U) is = tr(d j b i b k d l ) = b i, b k tr(d j d l ) = b i, b k d j, d l = δ ik δ jl. L = l ij b i d j with l ij := L, b i d j = b i, Ld j. (Why?) Remark 29 Suppose B := {e 1,..., e N } is an orthonormal basis for V. Then every L Lin(V) has the associated matrix [ˆl ij ] defined through N L = ˆlij e i e j. i,j=1 On the other hand, L also has the associated matrix representation [L] B through the relation [Lv] B = [L] B [v] B which must hold for all v V with = [l ij ] defined l ij = e i, Le j. It is straight forward to show (How?) that these two matrices are equal, i.e. [L] B = [l ij ] = [ˆl ij ] General Tensor Product Bases for Lin(V) and Their Duals Let B = {f 1,..., f N } be a general basis for (V,, ) with associated reciprocal basis B = {f 1,..., f N }. Then Lin(V) has four associated tensor product bases and corresponding reciprocal bases given by {f i f j } {f i f j } (43) {f i f j } {f i f j } (44) {f i f j } {f i f j } (45) {f i f j } {f i f j } (46) 21

22 where i, j = 1,..., N in each set. Every L Lin(V) has four matrix representations defined through L = = = = N ˆlij f i f j (Pure Contravariant) (47) i,j=1 N ˆlij f i f j (Pure Covariant) (48) i,j=1 N ˆli j f i f j (Mixed Contravariant-Covariant) (49) i,j=1 N i,j=1 ˆl j i f i f j (Mixed Covariant-Contravariant). (50) It follows easily (How?) that one can compute these various matrix components of L from the formulae ˆlij = f i, Lf j ˆlij = f i, Lf j ˆli j = f i, Lf j ˆl j i = f i, Lf j. Exercise 4 Let L = I, the identity transformation on V. Then I has the four matrix representations (47,48,49,50) with ˆlij = f i, f j = f ij ˆlij = f i, f j = f ij ˆli j = f i, f j = δj i j ˆl i = f i, f j = δj i where δ i j denotes the usual Kronecker symbol. Thus, the matrix for the identity transformation when using a general (non-orthonormal) basis has the accustomed form only using a mixed covariant and contravariant representation. Remark 30 Equivalent Matrix Representations. Given a general basis B = {f 1,..., f N } and its dual basis B = {f 1,..., f N }, one also has the four matrix representations of a linear transformation L Lin(V) defined through [w] B = [Lv] B = [L] (B,B) [v] B [w] B = [Lv] B = [L] (B,B )[v] B [w] B = [Lv] B = [L] (B,B )[v] B [w] B = [Lv] B = [L] (B,B)[v] B 22

23 which in component form becomes w i = l i jv j w i = l j i v j w i = l ij v j w i = l ij v j. One now readily shows that ˆlij = l ij, ˆlij = l ij, ˆli j = l i j and ˆl j i = l j i. Indeed, consider the following calculation: Lv = i,j ˆlij f i f j ( vk f k) = i,j ˆlij f i v k f j, f k = i,j = i ˆlij v j ) ˆlij f i v k δ k j f i ( j = i w i f i where w i = ˆl ij v j. It now follows from the definition of l ij that ˆl ij = l ij. The remaining identities can be deduced by similar reasoning. Henceforth, the matrices [ˆl ij ], [l ij ], etc, will be used interchangeably. Remark 31 Raising and Lowering Indices. The metric tensor F = [f ij ] = [ f i, f j ] and its dual counterpart [f ij ] = [ f i, f j ] can be used to switch between covariant and contravariant component forms of vectors and linear transformations (and indeed, tensors of all orders). This process is called raising and lowering indices in classical tensor algebra. Thus, for example, if v = v i f i = v i f i, then v i = v, f i = v k f k, f i = f ik v k. Thus, [f ij ] is used to transform the covariant components [v i ] of a vector v into the contravariant components [v i ], a process called raising the index. By similar reasoning one shows that v i = f ik v k, that is, the metric tensor F = [f ij ] is used to transform the contravariant components of a vector into the covariant components, a process called lowering the index. 23

24 Analogous raising/lowering index formulas hold for linear transformations. In particular, suppose w = Lv and consider the string of identities Lv = ij l ij f i f j ( v k f k ) = ij l ij f i v k f j, f k = i f i ( k l ik f kj ) v j. From the definition of l i j, it follows that l i j = l ik f kj. Thus, the tensor [f ij ] can be used to lower the column index of [l ij ] by right multiplication of matrices. Similarly, [f ij ] can be used to raise the column index by right multiplication. Moreover, they can be used to lower and raise the row index of the matrices [l ij ] as illustrated by the following string of identities. L = ij l ij f i f j = ij l ij (f iq f q ) f j = qj ( i f iq l ij ) f q f j which from the definition of the matrix [l j q ] implies that l j i = f ik l kj. (51) The following identities are readily obtained by similar arguments. l ij = f ik l j k = li kf ik, l ij = f ik l i j = l k i f ik, etc.... Applying (51) to the identity transformation for which l i j = δj, i l j i l ij = f ij, one sees that δ j i = f ikf kj. = δ j i, lij = f ij and It follows that the metric tensor [f ij ] and the matrix [f ij ] are inverses of one another, i.e. [f ij ] = [f ij ] 1. 24

25 5.1.6 Eigenvalues, Eigenvectors and Eigenspaces Invariance principles are central tools in most areas of mathematics and they come in many forms. In linear algebra, the most important invariance principle resides in the notion of an Eigenvector. Definition 27 Let V be a vector space and L Lin(V) a linear transformation on V. Then a non-zero vector v V is called an Eigenvector and λ F its associated Eigenvalue provided Lv = λv. (52) From the definition (52) it follows that L(span(v)) span(v), that is the line spanned by the eigenvector v is invariant under L in that L maps the line back onto itself. The set of all eigenvectors corresponding to a given eigenvalue λ (with the addition of the zero vector) forms a subspace of V (Why?), denoted E(λ), called the the Eigenspace corresponding to the eigenvalue λ. One readily shows (How?) from the definition (52) that eigenvectors corresponding to distinct eigenvalues of a given linear transformation L Lin(V) are independent, from which one concludes Re-writing the definition (52) as E(λ 1 ) E(λ 2 ) = {0} if λ 1 λ 2. (53) (L λi)v = 0 one sees that v is an eigenvector with eigenvalue λ if and only if v Nul(L λi). It follows that λ is an eigenvalue for L if and only if det(l λi) = 0. Recalling that det(l λi) is a polynomial of degree N, one defines the Characteristic Polynomial p L (λ) of the transformation L to be p L (λ) := det(l λi) = ( λ) N + i 1 (L)( λ) N i N 1 (L)( λ) + i N (L). (54) The coefficients I L := {i 1 (L),..., i N (L)} of the characteristic polynomial are called the Principal Invariants of the transformation L because they are invariant under similarity transformation. More specifically, let T Lin(V) be non-singular. Then the characteristic polynomial for T 1 LT is identical to that for L as can be seen from the calculation p T 1 LT(λ) = det(t 1 LT λi) = det(t 1 LT λt 1 T) = det(t 1 (L λi)t) = det(l λi) = p L (λ). 25

26 An explanation of why the elements of I L are called the Principal Invariants of L must wait until the upcoming section on spectral theory. The principle invariants i 1 (L) and i N (L) have simple forms. Indeed one can readily show (How?) that i 1 (L) = tr(l) and i N (L) = det(l). Moreover, if the characteristic polynomial has the factorization then one also has that p L (λ) = N (λ j λ), j=1 i 1 (λ) = tr(l) = λ λ N and i N (L) = det(l) = λ 1... λ N. 5.2 Classes of Linear Transformations An understanding of general linear transformations in Lin(V ) can be gleaned from a study of special classes of transformations and representation theorems expressing general linear transformations as sums or products of transformations from the special classes. The discussion is framed within the setting of a finite dimensional, inner product vector space. The first class of transformations introduced are the projections Projections Definition 28 A linear transformation P on (V,, ) is called a Projection provided P 2 = P. (55) Remark 32 It follows readily from the definition (55) that P is a projection if and only if I P is a projection. Indeed, assume P is a projection, then (I P ) 2 = I 2 2P + P 2 = I 2P + P = I P which shows that I P is a projection. The reverse implication follows by a similar argument since if I P is a projection, then one need merely write P = I (I P ) and apply the above calculation. Remark 33 The geometry of projections is illuminated by the observation that the there is a simple relationship between their range and nullspace. In particular, one can show easily from the definition (55) that Ran(P ) = Nul(I P ) and Nul(P ) = Ran(I P ). (56) 26

27 For example, to prove that Ran(P ) = Nul(I P ), let v Ran(P ). Then v = P u for some u V and (I P )v = (I P )P u = P u P 2 u = P u P u = 0. It follows that v Nul(P ) and hence that Ran(P ) Nul(I P ). (57) Conversely, if v Nul(I P ), then (I P )v = 0, or equivalently v = P v. Thus, v Ran(P ) from which it follows that Nul(I P ) Ran(P ). (58) From (57,58) one concludes that Ran(P ) = Nul(I P ) as required. The second equality in (56) follows similarly. Remark 34 Another important observation concerning projections is that V is a direct sum of the subspaces Ran(P ) and Nul(P ) To see (59), one merely notes that for any v V V = Ran(P ) Nul(P ). (59) v = P v + (I P )v, with P v Ran(P ) and (I P )v Nul(P ). This fact and (9) show (59). Because of (59) one often refers to a projection P as projecting V onto Ran(P ) along Nul(P ). An important subclass of projections are the Perpendicular Projections. Definition 29 A projection P is called a Perpendicular Projection provided Ran(P ) Nul(P ). A non-obvious fact about perpendicular projections is the Theorem 8 A projection P is a perpendicular projection if and only if P is self-adjoint, i.e. P = P. Proof: First assume P is self-adjoint and let v Ran(P ) and w Nul(P ). Then, v = P u for some u V and v, w = P u, w = u, P w = u, P w = u, 0 = 0 from which it follows that v w and hence that Ran(P ) Nul(P ). Conversely, suppose P is a perpendicular projection. One must show that P is selfadjoint, i.e. that P v, u = v, P u for all v, u V. (60) To that end, notice that P v, u = P v, P u + (I P )u = P v, P u + P v, (I P )u = P v, P u. (Why?) Since P v, P u is symmetric in v and u, the same must be true for P v, u, from which ones concludes (60). Question: Among the class of elementary tensor products, which (if any) are projections? Which are perpendicular projections? 27

28 Answer: An elementary tensor product a b is a projection if and only if a, b = 1. It is a perpendicular projection if and only if b = a and a = 1. (Why?) Thus, if a is a unit vector, then a a is perpendicular projection onto the line spanned by a. More generally, if a and b are perpendicular, unit vectors, then a a + b b is a perpendicular projection onto the plane (2-dimensional subspace) spanned by a and b. (Exercise.) Generalizing further, if a 1,..., a K are pairwise orthogonal unit vectors, then a 1 a a K a K is a perpendicular projection onto span(a 1,..., a K ) Diagonalizable Transformations A linear transformation L on a finite dimensional vector space V is called Diagonalizable provided it has a matrix representation that is diagonal, that is, there is a basis B for V such that [L] B = diag(l 1,..., l N ). (61) Notation: An N N diagonal matrix will be denoted A = diag(a 1,..., a N ) where A = [a ij ] with a 11 = a 1,..., a NN = a N and a ij = 0 for i j. The fundamental result concerning diagonalizable transformations is Theorem 9 A linear transformation L on V is diagonalizable if and only if V has a basis consisting entirely of eigenvectors of L. Proof: Assume first that V has a basis B = {f 1,..., f N } consisting of eigenvectors of L, that is Lf j = λ j f j for j = 1,..., N. Recall that the column N-tuples comprising the matrix [L] B are given by [Lf j ] B, the coordinates with respect to B of the vectors Lf j for j = 1,..., N. But 0 λ 1 [Lf 1 ] B = [λ 1 f 1 ] B = 0., [Lf 2] B = [λ 2 f 2 ] B = 0 It now follows immediately that λ , etc. [L] B = diag(λ 1,..., λ N ). (Why?) Conversely, suppose L is diagonalizable, that is, there exists a basis B = {f 1,..., f N } for V such that [L] B = diag(λ 1,..., λ N ). But then for each j = 1,..., N 0. 0 [Lf j ] B = [L] B [f j ] B = diag(λ 1,..., λ N ) 1 (j th -component=1) = [λ j f j ] B It follows that for each j = 1,..., N, Lf j = λ j f j (Why?), and hence that B = {f 1,..., f N } is a basis of eigenvectors of L. 28

29 5.2.3 Orthogonal and Unitary Transformations Let (V,, ) be a finite dimensional, inner product vector space. The the class of Orthogonal Transformations, in the case of a vector space over R, and the class of Unitary Transformation, in the case of a vector space over C, is defined by Definition 30 A transformation Q Lin(V) is called Orthogonal (real case) or Unitary (complex case) provided That is, the inverse of Q equals the adjoint of Q. Q 1 = Q or equivalently I = Q Q. (62) Remark 35 It follows immediately from (62) that orthogonal transformations preserve the inner product on V and hence leave lengths and angles invariant. Specifically, one has In particular, from (63) it follows that showing that Q preserves lengths. Remark 36 shows that Qa, Qb = a, Q Qb = a, Q 1 Qb = a, b. (63) Qa 2 = Qa, Qa = a, a = a 2 The class of orthogonal transformations will be denoted Orth. One easily It is useful to define the subclasses det(q) = ±1. (Why?) Orth + := {Q Orth det(q) = 1} and Orth := {Q Orth det(q) = 1}. Example 14 If V is the Euclidean plane R 2, then Orth + (R 2 ) is the set of (rigid) Rotations, i.e. given Q Orth + (R 2 ), there exists a 0 < θ < 2π such that ( ) cos(θ) sin(θ) Q =. (64) sin(θ) cos(θ) Also, the transformation R M := ( ) is in Orth (R 2 ) (Why?) (How does R M act geometrically?). More generally, Orth (R 2 ) = {R Orth(R 2 ) R = R M Q for some Q Orth + (R 2 )}. (Why?) Example 15 If V is the Euclidean space R 3, then Orth + is called the set of Proper Orthogonal Transformations or the set of Rotations. To motivate calling Q Orth + (R 3 ), one can prove that 29