12 CHAPTER 1. PRELIMINARIES Lemma 1.3 (Cauchy-Schwarz inequality) Let (; ) be an inner product in < n. Then for all x; y 2 < n we have j(x; y)j (x; x)

Transcription

1 1.4. INNER PRODUCTS,VECTOR NORMS, AND MATRIX NORMS 11 The estimate ^ is unbiased, but E(^ 2 ) = n?1 n 2 and is thus biased. An unbiased estimate is ^ 2 = 1 (x i? ^) 2 : n? 1 In x?? we show that the linear least squares problem arises out of a maximum likelihood estimation problem Inner Products,Vector Norms, and Matrix Norms We briey review basic results for inner products, vector norms, and matrix norms. In x1.4.1 basic properties of norms and inner products are given. In x1.4.2, we discusses the properties of the norms that are based on the Euclidean norm Basic Denitions and Lemmas First, we give the denition of an inner product. Denition An inner product in < n is a function (; ) mapping < n < n into < n that satises the following four axioms 1. (x; x) 0; (x; x) = 0; if and only if x = 0; x 2 < n, 2. (x; y) = (x; y); x; y 2 < n ; 2 <, 3. (x; y) = (y; x); x; y 2 < n, 4. (x + z; y) = (x; y) + (z; y); x; y; z 2 < n. We note that if we dene an inner product for complex vectors, the third axiom becomes (x; y) = (y; x) x; y 2 C n (1.5) where a denotes the complex conjugate of a. The inner product that is used most often is the Euclidean dot product (x; y) = x T y = x i y i : Others are introduced in Chapter??. The Cauchy-Schwarz inequality given below is quite useful for all inner products.

2 12 CHAPTER 1. PRELIMINARIES Lemma 1.3 (Cauchy-Schwarz inequality) Let (; ) be an inner product in < n. Then for all x; y 2 < n we have j(x; y)j (x; x) 1=2 (y; y) 1=2 : (1.6) Moreover, equality in (1.6) holds if and only if x = y for some 2 <. This inequality leads to the following denition of the angle between two vectors relative to an inner product. Denition The angle between the two nonzero vectors x; y 2 < n with respect to an inner product (; ) is given by (x; y) cos = (x; x) 1=2 (y; y) 1=2 : (1.7) The next important denition is that of a vector norm. Denition A norm in < n is a function k k mapping < n into < satisfying the following three axioms 1. kxk 0; kxk = 0 if and only if x = 0; x 2 < n. 2. kxk = jjkxk; x 2 < n ; 2 <. 3. kx + yk kxk + kyk; x; y 2 < n : Denition and the Cauchy-Schwarz inequality give us that for any inner product (; ) we can dene a norm k k by kxk = (x; x) 1=2 : (1.8) The most well known of these norms is the Euclidean norm given by! 1=2 kxk2 =? x T x 1=2 = n X x 2 i : (1.9) It is also referred to as the two-norm. This is the most important vector norm in the book. The subscript \2" is explained below. The two-norm is one of the class of p-norms. These are given by kxk p = ( jx i j p ) 1=p ; p 1: (1.10)

3 1.4. INNER PRODUCTS,VECTOR NORMS, AND MATRIX NORMS 13 Clearly, p = 2 leads to the denition (1.9). Except for the two-norm, the only norms we will use from this class are the one-norm given by and the 1-norm given by kxk1 = jx i j; (1.11) kxk 1 = max 1in jx ij = lim p!1 kxk p: (1.12) The next lemma, the Holder inequality, relates the p-norms to the Euclidean inner product. Lemma 1.4 (Holder inequality) Let k k p and k k q be norms in < n from the class (1.10) such that p?1 + q?1 = 1. Then for all x; y 2 < n we have jx T yj kxk p kyk q : (1.13) The interesting cases for us are p = q = 2, p = 1; q = 1, and p = 1; q = 1. For p = q = 2, the Holder inequality is just the Cauchy-Schwarz inequality. The next inequality states that if we can bound kxk, for a given x 2 < n in any norm kk, the quantity kxk can be bounded in any other norm kk. Lemma 1.5 All norms on < n are uniformly equivalent, meaning that for any two norms k k and k k there are constants c1 and c2 such that for all x 2 < n. c1kxk kxk c2kxk (1.14) For the two-norm, the one-norm, and the 1?norm the uniform equivalence relations are summarized by 1 p n kxk1 kxk2 kxk1; (1.15) We will also need norms for matrices. kxk 1 kxk2 p nkxk 1 ; (1.16) kxk 1 kxk1 nkxk 1 : (1.17) Denition A norm in < mn is a function k k mapping < mn into < satisfying the following three axioms

4 14 CHAPTER 1. PRELIMINARIES 1. kxk 0; kxk = 0 if and only if X = 0; X 2 < mn 2. kxk = jjkxk X 2 < mn ; 2 < 3. kx + Y k kxk + ky k X; Y 2 < mn. This denition is isomorphic to the denition of a vector norm on < mn. For example, the Frobenius norm dened by kxk F = m X x 2 ij j=1 1 A 1=2 is isomorphic to the two-norm on < mn. For semantic reasons, we dene a family of norms. (1.18) Denition Let k k ;m;n be a norm on < mn and let it be well-dened for every positive integer m and n. Then the set N = fk k ;m;n : m; n positive integers g is called a family of norms. For any positive integers m and n, and any X 2 < mn, we denote kxk ;m;n by kxk. That is, for any matrix X, the quantity kxk is the appropriate member of N applied to X. The set V N given by V = fk k ;m;1 : m positive integer g is the associated family of vector norms. For any x 2 < m, the quantity kxk = k(x)k is the appropriate member of V applied to x. Since X represents a linear operator from < n to < m, it is appropriate to dene the induced norm k k on < mn by It is a simple matter to show that kxyk kxk = sup : (1.19) y6=0 kyk kxk = max kyk =1 kxyk : (1.20) Note that the maximum is taken over a closed, bounded set, thus we have that kxk = kxy k (1.21)

5 1.4. INNER PRODUCTS,VECTOR NORMS, AND MATRIX NORMS 15 for some y such that ky k = 1. The above denition leads to the very useful bound kxyk kxk kyk (1.22) where equality occurs for every vector of the form y ; 2 <. If X 2 < m1 then (1.22) appears to give a second denition for kxk. However, it is easily veried that if X = (x), for some x 2 < m, then kxk, dened from the matrix norm (1.22), and kxk, dened from the vector norm, are the same value. Thus the notation k k is unambiguous for all m and n. For any induced norm k k, the identity matrix I n for < nn satises However, for the Frobenius norm ki n k = 1: (1.23) ki n k F = p n; thus it is not an induced norm for any vector norm. For the one-norm and the 1-norm there are formulas for the corresponding matrix norms and for a vector y satisfying (1.21). The one-norm formula is kxk1 = max 1jn If j max is the index of a column such that mx mx kxk1 = jx i;jmax j jx ij j: (1.24) then y = e jmax, the corresponding column of the identity matrix. The 1-norm formula is kxk 1 = max If i max is the index of a row such that 1im kxk 1 = j=1 jx ij j: (1.25) j=1 jx imax;jj then the vector y = (y 1 ; : : : ; y n) T with components y j = sign(x imax;j) satises (1.21). Note that kxk 1 = kx T k1.

6 16 CHAPTER 1. PRELIMINARIES The matrix two-norm does not have a formula like (1.24) or (1.25) and all other formulations are really equivalent to (1.19). Moreover, computing the vector y in (1.21) is a nontrivial task that we will discuss in Chapters?? and??. The induced norms have a convenient property that is important in understanding matrix computations. For X 2 < mn and Y 2 < ns consider kxy k. We have that Thus kxy k = max kzk =1 kxy zk max kzk =1 kxk ky zk = kxk max kzk =1 ky zk = kxk ky k : kxy k kxk ky k : (1.26) A norm k k (or really family of norms) that satises the property (1.26) is said to be consistent. Since they are induced norms the two-norm, one-norm, and the 1-norm are all consistent. The Frobenius norm also satises (1.26). An example of a matrix norm that is not consistent is given below. Example 1.1 Consider the norm k k on < mn given by kxk = max (i;j) jx ijj: This is simply the 1-norm applied to X written out as vector in < mn. For m = n = 2, consider 1 1 X = Y = : 1 1 Note that 2 2 XY = 2 2 and thus kxy k = 2 > kxk ky k = 1. Clearly, kk norm is not consistent. Henceforth, we use only consistent families of norms. The above dened matrix norms satisfy the following useful inequalities for all X 2 < mn and Y 2 < ns : (mn)? 1 4 (kxk 1kXk 1 ) 1=2 kxk2 (kxk1kxk 1 ) 1=2 ; (1.27) 1 1 max pn ; p kxk F kxk2 kxk F ; (1.28) m kxy k F kxk2ky k F ; (1.29) kxy k F kxk F ky k2: (1.30)

7 1.4. INNER PRODUCTS,VECTOR NORMS, AND MATRIX NORMS 17 For X 6= 0, the lower bound in (1.28) can be tightened into 1 p rank(x) kxk F kxk2: (1.31) Since rank(x) is often more expensive to compute than a good approximation to kxk2, the practical use of (1.31) is limited. For any diagonal matrix = diag() and any p-norm k k p kk p = kk 1 = max 1in j ij 1 p 1: For any matrix X 2 < mn we have the following relations for the matrix jxj: kxk2 k jxj k2 p rank(x)kxk2 minf p m; p ngkxk2; kxk1 = k jxj k1; kxk 1 = k jxj k 1 ; kxk F = k jxj k F : In the context of linear least squares, our interest will be in the two-norm or in the Frobenius norm. The other norms will be used to bound them. Some special properties of these norms are given in the next section The Two-Norm, the Frobenius Norm, and Orthogonality We begin by dening orthogonality. We then relate orthogonality to the matrix two-norm and Frobenius norm. Denition Two vectors x; y 2 < n are orthogonal with respect to an inner product (; ) if (x; y) = 0. The set S1 < n is orthogonal to the set S2 < n, if for each x 2 S1 and y 2 S2, we have (x; y) = 0. We write x? y; S1? S2 to mean \x is orthogonal to y and \S1 is orthogonal to S2." The zero vector is orthogonal to any vector and the set f0g is orthogonal to any set. An orthogonal set of vectors is dened as follows. Denition A set of k vectors fx1; x2; : : : ; x k g, where each x i 2 < n, is said to be an orthogonal with respect to the inner product (; ) if (x i ; x j ) = 0 for i 6= j. The set is said to be orthonormal if it is orthogonal and (x i ; x i ) = 1 for i = 1; 2; : : : ; k The denition of an orthogonal matrix is related to the denition for vectors.