Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases

Size: px
Start display at page:

Download "Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases"

Transcription

1 Week Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases. Opening Remarks.. Low Rank Approximation 38

2 Week. Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases 38.. Outline.. Opening Remarks Low Rank Approximation Outline What You Will Learn Projecting a Vector onto a Subspace Component in the Direction of An Application: Rank- Approximation Projection onto a Subspace An Application: Rank- Approximation An Application: Rank-k Approximation Orthonormal Bases The Unit Basis Vectors, Again Orthonormal Vectors Orthogonal Bases Orthogonal Bases Alternative Explanation The QR Factorization Solving the Linear Least-Squares Problem via QR Factorization The QR Factorization Again Change of Basis The Unit Basis Vectors, One More Time Change of Basis Singular Value Decomposition The Best Low Rank Approximation Enrichment The Problem with Computing the QR Factorization QR Factorization Via Householder Transformations Reflections More on SVD Wrap Up Homework Summary

3 .. Opening Remarks What You Will Learn Upon completion of this unit, you should be able to Given vectors a and b in R m, find the component of b in the direction of a and the component of b orthogonal to a. Given a matrix A with linear independent columns, find the matrix that projects any given vector b onto the column space A and the matrix that projects b onto the space orthogonal to the column space of A, which is also called the left null space of A. Understand low rank approximation, projecting onto columns to create a rank-k approximation. Identify, apply, and prove simple properties of orthonormal vectors. Determine if a set of vectors is orthonormal. Transform a set of basis vectors into an orthonormal basis using Gram-Schmidt orthogonalization. Compute an orthonormal basis for the column space of A. Apply Gram-Schmidt orthogonalization to compute the QR factorization. Solve the Linear Least-Squares Problem via the QR Factorization. Make a change of basis. Be aware of the existence of the Singular Value Decomposition and that it provides the best rank-k approximation.

4 Week. Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases 384. Projecting a Vector onto a Subspace.. Component in the Direction of... Consider the following picture: z = χa a w b Span{a} = Ca Here, we have two vectors, a,b R m. They exist in the plane defined by Span{a,b} which is a two dimensional space unless a and b point in the same direction. From the picture, we can also see that b can be thought of as having a component z in the direction of a and another component w that is orthogonal perpendicular to a. The component in the direction of a lies in the Span{a} = Ca here a denotes the matrix with only once column, a while the component that is orthogonal to a lies in Span{a}. Thus, b = z + w, where z = χa with χ R; and a T w =. Noting that w = b z we find that or, equivalently, = a T w = a T b z = a T b χa a T aχ = a T b. We have seen this before. Recall that when you want to approximately solve Ax = b where b is not in CA via Linear Least Squares, the best solution satisfies A T Ax = A T b. The equation that we just derived is the exact same, except that A has one column: A = a. Then, provided a, Thus, the component of b in the direction of a is given by χ = a T a a T b. u = χa = a T a a T ba = aa T a a T b = [ aa T a a T ] b.

5 .. Projecting a Vector onto a Subspace 385 Note that we were able to move a to the left of the equation because a T a and a T b are both scalars. The component of b orthogonal perpendicular to a is given by w = b z = b aa T a a T b = Ib aa T a a T b = I aa T a a T b. Summarizing: z = aa T a a T b is the component of b in the direction of a; and w = I aa T a a T b is the component of b perpendicular orthogonal to a. We say that, given vector a, the matrix that projects any given vector b onto the space spanned by a is given by aa T a a T = a T a aat since aa T a a T b is the component of b in Span{a}. Notice that this is an outer product: a a T a a T. v T We say that, given vector a, the matrix that projects any given vector b onto the space orthogonal to the space spanned by a is given by I aa T a a T = I a T a aat = I av T, since I aa T a a T b is the component of b in Span{a}. Notice that I a T a aat = I av T is a rank- update to the identity matrix. Homework... Let a = and P a x and P a x be the projection of vector x onto Span{a} and Span{a}, respectively. Compute. P a. Pa 3. P a 4 4. Pa 4 = = = = 5. Draw a picture for each of the above.

6 Week. Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases 386 Homework... Let a = Span{a}, respectively. Compute. P a =. Pa = 3. P a = 4. Pa = and P ax and P a x be the projection of vector x onto Span{a} and Homework...3 Let a,v,b R m. What is the approximate cost of computing av T b, obeying the order indicated by the parentheses? m + m. 3m. m + 4m. What is the approximate cost of computing v T ba, obeying the order indicated by the parentheses? m + m. 3m. m + 4m. For computational efficiency, it is important to compute aa T a a T b according to order indicated by the following parentheses: a T a a T ba. Similarly, I aa T a a T b should be computed as b a T a a T ba.

7 .. Projecting a Vector onto a Subspace 387 Homework...4 Given a,x R m, let P a x and P a x be the projection of vector x onto Span{a} and Span{a}, respectively. Then which of the following are true:. P a a = a. True/False. P a χa = χa. True/False 3. P a χa = the zero vector. True/False 4. P a P a x = P a x. True/False 5. P a P a x = P a x. True/False 6. P a P a x = the zero vector. True/False Hint: Draw yourself a picture... An Application: Rank- Approximation Consider the picture b j β i, j This picture can be thought of as a matrix B R m n where each element in the matrix encodes a pixel in the picture. The jth column of B then encodes the jth column of pixels in the picture. Now, let s focus on the first few columns. Notice that there is a lot of similarity in those columns. This can be illustrated by plotting the values in the column as a function of the element in the column:

8 Week. Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases 388 In the graph on the left, we plot β i, j, the value of the i, j pixel, for j =,,,3 in different colors. The picture on the right highlights the columns for which we are doing this. The green line corresponds to j = 3 and you notice that it is starting to deviate some for i near 5. If we now instead look at columns j =,,,, where the green line corresponds to j =, we see that that column in the picture is dramatically different: Changing this to plotting j =,,,3 and we notice a lot of similarity again:

9 .. Projecting a Vector onto a Subspace 389 Now, let s think about this from the point of view taking one vector, say the first column of B, b, and projecting the other columns onto the span of that column. What does this mean? Partition B into columns B = b b b n. Pick a = b. Focus on projecting b onto Span{a}: aa T a a T b = aa T a a T a Since b = a = a. Of course, this is what we expect when projecting a vector onto itself. Next, focus on projecting b onto Span{a}: since b is very close to b. aa T a a T b Do this for all columns, and create a picture with all of the projected vectors: aa T a a T b aa T a a T b aa T a a T b Now, remember that if T is some matrix, then T B = T b T b T b. If we let T = aa T a a T the matrix that projects onto Span{a}, then aa T a a T b b b = aa T a a T B. We can manipulate this further by recognizing that y T = a T a a T B can be computed as y = a T a B T a: aa T a a T B = a a T a B T a y T = ay T We now recognize ay T as an outer product a column vector times a row vector.

10 Week. Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases 39 If we do this for our picture, we get the picture on the left: Notice how it seems like each column is the same, except with some constant change in the gray-scale. The same is true for rows. Why is this? If you focus on the left-most columns in the picture, they almost look correct comparing to the left-most columns in the picture on the right. Why is this? The benefit of the approximation on the left is that it can be described with two vectors: a and y n + m floating point numbers while the original matrix on the right required an entire matrix m n floating point numbers. The disadvantage of the approximation on the left is that it is hard to recognize the original picture... Homework... Let S and T be subspaces of R m and S T. dims dimt. Always/Sometimes/Never Homework... Let u R m and v R n. Then the m n matrix uv T has a rank of at most one. True/False Homework...3 Let u R m and v R n. Then uv T has rank equal to zero if Mark all correct answers.. u = the zero vector in R m.. v = the zero vector in R n. 3. Never. 4. Always...3 Projection onto a Subspace No video this section Next, consider the following picture:

11 .. Projecting a Vector onto a Subspace 39 z = Ax b w CA What we have here are Matrix A R m n. The space spanned by the columns of A: CA. A vector b R m. Vector z, the component of b in CA which is also the vector in CA closest to the vector b. Since this vector is in the column space of A, z = Ax for some vector x R n. The vector w which is the component of b orthogonal to CA. The vectors b,z,w, all exist in the same planar subspace since b = z + w, which is the page on which these vectors are drawn in the above picture. Thus, b = z + w, where z = Ax with x R n ; and A T w = since w is orthogonal to the column space of A and hence in N A T. Noting that w = b z we find that or, equivalently, = A T w = A T b z = A T b Ax A T Ax = A T b. This should look familiar! Then, provided A T A exists which, we saw before happens when A has linearly independent columns, Thus, the component of b in CA is given by x = A T A A T b. z = Ax = AA T A A T b while the component of b orthogonal perpendicular to CA is given by Summarizing: w = b z = b AA T A A T b = Ib AA T A A T b = I AA T A A T b. z = AA T A A T b w = I AA T A A T b.

12 Week. Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases 39 We say that, given matrix A with linearly independent columns, the matrix that projects a given vector b onto the column space of A is given by AA T A A T since AA T A A T b is the component of b in CA. We say that, given matrix A with linearly independent columns, the matrix that projects a given vector b onto the space orthogonal to the column space of A which, recall, is the left null space of A is given by I AA T A A T since I AA T A A T b is the component of b in CA = N A T. Homework..3. Consider A = 4 and b = 7.. Find the projection of b onto the column space of A.. Split b into z + w where z is in the column space and w is perpendicular orthogonal to that space. 3. Which of the four subspaces CA, RA, N A, N A T contains w? For computational reasons, it is important to compute AA T A A T x according to order indicated by the following parentheses: A[A T A [A T x]] Similarly, I AA T A A T x should be computed as x [A[A T A [A T x]]]..4 An Application: Rank- Approximation Earlier, we took the first column as being representative of all columns of the picture. Looking at the picture, this is clearly not the case. But what if we took two columns instead, say column j = and j = n/, and projected each of the columns onto the subspace spanned by those two columns:

13 .. Projecting a Vector onto a Subspace 393 b b n/ Partition B into columns B = Pick A = a a = b b b n. b b n/. Focus on projecting b onto Span{a,a } = CA: AA T A A T b = a = b because a is in CA and a is therefore the best vector in CA. Next, focus on projecting b onto Span{a}: since b is very close to a. AA T A A T b b Do this for all columns, and create a picture with all of the projected vectors: AA T A A T b AA T A A T b AA T A A T b Now, remember that if T is some matrix, then T B = T b T b T b. If we let T = AA T A A T the matrix that projects onto CA, then AA T A A T b b b = AA T A A T B.

14 Week. Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases 394 We can manipulate this by letting W = B T AA T A so that A A T A A T B W T = AW T. Notice that A and W each have two columns. We now recognize AW T is the sum of two outer products: AW T = a a w w T = a a wt w T = a w T + a w T. It can be easily shown that this matrix has rank of at most two, which is why this would be called a rank- approximation of B. If we do this for our picture, we get the picture on the left: We are starting to see some more detail. We now have to store only a n and m matrix A and W...5 An Application: Rank-k Approximation Rank-k approximations We can improve the approximations above by picking progressively more columns for A. The following progression of pictures shows the improvement as more and more columns are used, where k indicates the number of columns:

15 .. Projecting a Vector onto a Subspace 395 k= k= k = k = 5 k = 5 original Homework..5. Let U Rm k and V Rn k. Then the m n matrix UV T has rank at most k. True/False * View at edx

16 Week. Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases 396 Homework..5. We discussed in this section that the projection of B onto the column space of A is given by AA T A A T B. So, if we compute V = A T A A T B, then AV is an approximation to B that requires only m k matrix A and k n matrix V. To compute V, we can perform the following steps: Form C = A T A. Compute the LU factorization of C, overwriting C with the resulting L and U. Compute V = A T B. Solve LX = V, overwriting V with the solution matrix X. Solve UX = V, overwriting V with the solution matrix X. Compute the approximation of B as A V A times V. In practice, you would not compute this approximation, but store A and V instead, which typically means less data is stored. To experiments with this, download Week.zip, place it in LAFF-.xM -> Programming and unzip it. Then examine the file Week/CompressPicture.m, look for the comments on what operations need to be inserted, and insert them. Execute the script in the Command Window and see how the picture in file building.png is approximated. Play with the number of columns used to approximate. Find your own picture! It will have to be a black-and-white picture for what we discussed to work. Notice that A T A is a symmetric matrix, and it can be shown to be symmetric positive definite under most circumstances when A has linearly independent columns. This means that instead of the LU factorization, one can use the Cholesky factorization see the enrichment in Week 8. In Week.zip you will also find a function for computing the Cholesky factorization. Try to use it to perform the calculations..3 Orthonormal Bases.3. The Unit Basis Vectors, Again Recall the unit basis vectors in R 3 : e =, e = and e =. This set of vectors forms a basis for R 3 ; they are linearly independent and any vector x R 3 can be written as a linear combination of these three vectors. Now, the set v = is also a basis for R 3, but is not nearly as nice: Two of the vectors are not of length one., v = and v =

17 .3. Orthonormal Bases 397 They are not orthogonal to each other. There is something pleasing about a basis that is orthonormal. By this we mean that each vector in the basis is of length one, and any pair of vectors is orthogonal to each other. A question we are going to answer in the next few units is how to take a given basis for a subspace and create an orthonormal basis from it. Homework.3.. Consider the vectors v =, v = and v =. Compute a v T v = b v T v = c v T v =. These vectors are orthonormal. True/False.3. Orthonormal Vectors Definition. Let q,q,...,q k R m. Then these vectors are mutually orthonormal if for all i, j < k : q T if i = j i q j = otherwise. Homework cosθ sinθ sinθ cosθ cosθ sinθ sinθ 3. The vectors cosθ sinθ cosθ T cosθ sinθ T, sinθ cosθ cosθ sinθ sinθ cosθ sinθ cosθ = = are orthonormal. 4. The vectors sinθ, cosθ are orthonormal. cosθ sinθ True/False True/False

18 Week. Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases 398 Homework.3.. Let q,q,...,q k R m be a set of orthonormal vectors. Let Q = q q q k. Then Q T Q = I. TRUE/FALSE Homework.3..3 Let Q R m k with k m and Q T Q = I. Partition Q = q q q k. Then q,q,...,q k are orthonormal vectors. TRUE/FALSE Homework.3..4 Let q R m be a unit vector which means it has length one. Then the matrix that projects vectors onto Span{q} is given by qq T. True/False Homework.3..5 Let q R m be a unit vector which means it has length one. Let x R m. Then the component of x in the direction of q in Span{q} is given by q T xq. True/False Homework.3..6 Let Q R m n have orthonormal columns which means Q T Q = I. Then the matrix that projects vectors onto the column space of Q, CQ, is given by QQ T. True/False

19 .3. Orthonormal Bases 399 Homework.3..7 Let Q R m n have orthonormal columns which means Q T Q = I. Then the matrix that projects vectors onto the space orthogonal to the columns of Q, CQ, is given by I QQ T. True/False.3.3 Orthogonal Bases The fundamental idea for this unit is that it is convenient for a basis to be orthonormal. The question is: how do we transform a given set of basis vectors e.g., the columns of a matrix A with linearly independent columns into a set of orthonormal vectors that form a basis for the same space? The process we will described is known as Gram-Schmidt orthogonalization GS orthogonalization. The idea is very simple: Start with a set of n linearly independent vectors, a,a,...,a n R m. Take the first vector and make it of unit length: q = a / a ρ,, where ρ, = a, the length of a. Notice that Span{a } = Span{q } since q is simply a scalar multiple of a. This gives us one orthonormal vector, q. Take the second vector, a, and compute its component orthogonal to q : a = I q q T a = a q q T a = a q T a ρ, q. Take a, the component of a orthogonal to q, and make it of unit length: q = a / a ρ,, We will see later that Span{a,a } = Span{q,q }. This gives us two orthonormal vectors, q,q.

20 Week. Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases 4 Take the third vector, a, and compute its component orthogonal to Q = and hence Span{q,q } = CQ : a = I Q Q T a = a Projection onto CQ q q qt q T = a q T a q + q T a q = a q T a q Component in direction = a Q Q T a Component = a in CQ a = a q q q T a q. Component in direction q q orthogonal to both q and q q q qt a q T a q q T a of q of q Notice: a q T a q equals the vector a with the component in the direction of q subtracted out. a q T a q q T a q equals the vector a with the components in the direction of q and q subtracted out. Thus, a equals component of a that is orthogonal to both q and q. Take a, the component of a orthogonal to q and q, and make it of unit length: q = a / a We will see later that Span{a,a,a } = Span{q,q,q }. This gives us three orthonormal vectors, q,q,q. Continue repeating the process Take vector a k, and compute its component orthogonal to Q k = q q q k orthogonal to all vectors q,q,...,q k and hence Span{q,q,...,q k } = CQ k : a k = I Q k Q kt a k = a k Q k Q kt a k = a k = a k q q q k q T q T. ρ, a k = a k, q q q k q q T q k ak q T a k q T q q q k k. q T k = a k q T a k q q T a k q q T k a kq k. q T k a k Notice: a k q T a kq equals the vector a k with the component in the direction of q subtracted out. a k q T a kq q T a kq equals the vector a k with the components in the direction of q and q subtracted out. a k q T a kq q T a kq q T k a kq k equals the vector a k with the components in the direction of q,q,...,q k subtracted out.

21 .3. Orthonormal Bases 4 Thus, a k equals component of a k that is orthogonal to all vectors q j that have already been computed. Take a k, the component of a k orthogonal to q,q,...q k, and make it of unit length: q k = a k / a k ρ k,k, We will see later that Span{a,a,...,a k } = Span{q,q,...,q k }. This gives us k + orthonormal vectors, q,q,...,q k. Continue this process to compute q,q,...,q n. The following result is the whole point of the Gram-Schmidt process, namely to find an orthonormal basis for the span of a given set of linearly independent vectors. Theorem. Let a,a,...,a k R m be linearly independent vectors and let q,q,...,q k R m be the result of Gram- Schmidt orthogonalization. Then Span{a,a,...,a k } = Span{q,q,...,q k }. The proof is a bit tricky and in some sense stated in the material in this unit so we do not give it here..3.4 Orthogonal Bases Alternative Explanation We now give an alternate explanation for Gram-Schmidt orthogonalization. We are given linearly independent vectors a,a,...,a n R m and would like to compute orthonormal vectors q,q,...,q n R m such that Span{a,a,...,a n } equals Span{q,q,...,q n }. Let s put one more condition on the vectors q k : Span{a,a,...,a k } = Span{q,q,...,q k } for k =,,...,n. In other words, Span{a } = Span{q } Span{a,a } = Span{q,q }.. Span{a,a,...,a k } = Span{q,q,...,q k }. Span{a,a,...,a n } = Span{q,q,...,q n } Computing q Now, Span{a } = Span{q } means that a = ρ, q for some scalar ρ,. Since q has to be of length one, we can choose ρ, := a q := a /ρ,. Notice that q is not unique: we could have chosen ρ, = a and q = a /ρ,. This non-uniqueness is recurring in the below discussion, and we will ignore it since we are merely interested in a single orthonormal basis.

22 Week. Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases 4 Computing q Next, we note that Span{a,a } = Span{q,q } means that a = ρ, q + ρ, q for some scalars ρ, and ρ,. We also know that q T q = and q T q = since these vectors are orthonormal. Now so that q T a = q T ρ, q + ρ, q = q T ρ, q + q T ρ, q = ρ, q T q ρ, = q T a. Once ρ, has been computed, we can compute the component of a orthogonal to q : + ρ, q T q = ρ, ρ, q a = a q T a ρ, q after which a = ρ,q. Again, we can now compute ρ, as the length of a and normalize to compute q : ρ, := q T a a := a ρ, q ρ, := a q := a /ρ,. Computing q We note that Span{a,a,a } = Span{q,q,q } means that a = ρ, q + ρ, q + ρ, q for some scalars ρ,, ρ, and ρ,. We also know that q T q =, q T q = and q T q = since these vectors are orthonormal. Now q T a = q T ρ, q + ρ, q + ρ, q = ρ, q T q + ρ, q T q + ρ, q T q = ρ, so that ρ, = q T a. q T a = q T ρ, q + ρ, q + ρ, q = ρ, q T q + ρ, q T q + ρ, q T q = ρ, so that ρ, = q T a. Once ρ, and ρ, have been computed, we can compute the component of a orthogonal to q and q : ρ, q = a q T a q q T a q a ρ, ρ, after which a = ρ,q. Again, we can now compute ρ, as the length of a and normalize to compute q : ρ, := q T a ρ, := q T a a := a ρ, q ρ, q ρ, := a q := a /ρ,.

23 .3. Orthonormal Bases 43 Computing q k Let s generalize this: Span{a,a,...,a k } = Span{q,q,...,q k } means that k a k = ρ,k q + ρ,k q + + ρ k,k q k + ρ k,k q k = j= ρ j,k q j + ρ k,k q k for some scalars ρ,k,ρ,k,...,ρ k,k. We also know that q T i q j = if i = j otherwise. Now, if p < k, q T p a k = q T p k ρ j,k q j + ρ k,k q k = j= k j= ρ j,k q T p q j + ρ k,k q T p q k = ρ p,k q T p q p = ρ p,k so that ρ p,k = q T p a k. Once the scalars ρ p,k have been computed, we can compute the component of a k orthogonal to q,...,q k : ρ k,k q k a k k = a k j= q T j a k ρ j,k q j after which a k = ρ k,kq k. Once again, we can now compute ρ k,k as the length of a k and normalize to compute q k: ρ,k := q T a k. ρ k,k := q T k a k a k k := a k ρ k,k := a k j= q k := a k /ρ k,k. ρ j,k q j An algorithm The above discussion yields an algorithm for Gram-Schmidt orthogonalization, computing q,...,q n and all the ρ i, j s as a side product. This is not a FLAME algorithm so it may take longer to comprehend:

24 Week. Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases 44 for k =,...,n for p =,...,k ρ p,k := q T p a k endfor a k := a k for j =,...,k a k := a k ρ j,kq j endfor ρ k,k := a k q k := a k /ρ k,k endfor } } ρ,k ρ,k. ρ k,k q T a k q T = a k =. q T k a k a k = a k k j= ρ j,kq j = a k Normalize a k to be of length one. q T q T. q T k a k = q q q k q q q k T ak ρ,k ρ,k. ρ k,k Homework.3.4. Consider A = Compute an orthonormal basis for CA. Homework.3.4. Consider A =. Compute an orthonormal basis for CA. Homework Consider A = 4. Compute an orthonormal basis for CA..3.5 The QR Factorization Given linearly independent vectors a,a,...,a n R m, the last unit computed the orthonormal basis q,q,...,q n such that Span{a,a,...,a n } equals Span{q,q,...,q n }. As a side product, the scalars ρ i, j = q T i a j were computed, for i j. We now show that in the process we computed what s known as the QR factorization of the matrix A = a a a n : ρ, ρ, ρ,n ρ, ρ,n a a a n = q q q n A Q ρ n,n R

25 .3. Orthonormal Bases 45 Notice that Q T Q = I since its columns are orthonormal and R is upper triangular. In the last unit, we noticed that a = ρ, q a = ρ, q + ρ, q..... a n = ρ,n q + ρ,n q + + ρ n,n q n If we write the vectors on the left of the equal signs as the columns of a matrix, and do the same for the vectors on the right of the equal signs, we get a a a n A = = ρ, q ρ, q + ρ, q ρ,n q + ρ,n q + + ρ n,n q n q q q n Q ρ, ρ, ρ,n ρ, ρ,n ρ n,n Bingo, we have shown how Gram-Schmidt orthogonalization computes the QR factorization of a matrix A. Homework.3.5. Consider A =. Compute the QR factorization of this matrix. Hint: Look at Homework.3.4. Check that QR = A. Homework.3.5. Considerx!m A =. Compute the QR factorization of this matrix. 4 Hint: Look at Homework Check that A = QR. R.3.6 Solving the Linear Least-Squares Problem via QR Factorization Now, let s look at how to use the QR factorization to solve Ax b when b is not in the column space of A but A has linearly independent columns. We know that the linear least-squares solution is given by x = A T A A T b.

26 Week. Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases 46 Now A = QR where Q T Q = I. Then x = A T A A T b = QR = R T Q T Q I = R Q T b. A T QR A QR A T b R R T Q T b = R T R R T Q T b = R R T R T I Q T b Thus, the linear least-square solution, x, for Ax b when A has linearly independent columns solves Rx = Q T b. Homework.3.6. In Homework.3.4. you were asked to consider A = and compute an orthonormal basis for CA. In Homework.3.5. you were then asked to compute the QR factorization of that matrix. Of course, you could/should have used the results from Homework.3.4. to save yourself calculations. The result was the following factorization A = QR: = 3 Now, compute the best solution in the linear least-squares sense, ˆx, to This is the same problem as in Homework.4... u = Q T b = The solution to R ˆx = u is ˆx = χ = χ The QR Factorization Again We now give an explanation of how to compute the QR factorization that yields an algorithm in FLAME notation. We wish to compute A = QR where A,Q R m n and R R n n. Here Q T Q = I and R is upper triangular. Let s partition these matrices: A = A a A, Q = Q q Q, and R r R ρ r T R,

27 .3. Orthonormal Bases 47 where A,Q R m k and R R k k. Now, A = QR means that A a A = Q q Q R r R ρ r T R so that A a A = Q R Q r + ρ q Q R + q r T + Q R. Now, assume that Q and R have already been computed so that A = Q R. Let s focus on how to compute the next column of Q, q, and the next column of R, r ρ : a = Q r + ρ q implies that Q T a = Q T Q r + ρ q = Q T Q I r + ρ Q T q = r, since Q T Q = I the columns of Q are orthonormal and Q T q = q is orthogonal to all the columns of Q. So, we can compute r as r := Q T a. Now we can compute a, the component of a orthogonal to the columns of Q : a := a Q r = a Q Q T a = I Q Q T a, the component of a orthogonal to CQ. Rearranging a = Q r + ρ q yields ρ q = a Q r = a. Now, q is simply the vector of length one in the direction of a. Hence we can choose ρ := a q := a /ρ. All of these observations are summarized in the algorithm in Figure.

28 Week. Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases 48 Algorithm: [Q, R] := QRA, Q, R Partition A A L A R, Q where A L and Q L have columns, R T L is while na L < na do Repartition A L A R R T L R BL R T R R BR A a A, R r R r T ρ r T R r R Q L Q R, R R T L Q L Q R R BL R T R R BR Q q Q, r := Q T a a := a Q r ρ := a q = a /ρ Continue with A L A R R T L R BL endwhile R T R R BR A a A, R r R r T ρ r T R r R Q L Q R Q q Q, Figure.: QR facorization via Gram-Schmidt orthogonalization. Homework.3.7. Implement the algorithm for computing the QR factorization of a matrix in Figure. [ Q out, R out ] = QR unb A, Q, R where A and Q are m n matrices and R is an n n matrix. You will want to use the routines laff gemv, laff norm, and laff invscal. Alternatively, use native MATLAB operations. Store the routine in Test the routine with A = [ ]; Q = zeros 4, 3 ; R = zeros 3, 3 ; [ Q_out, R_out ] = QR_unb A, Q, R ; Next, see if A = QR: A - Q_out * R_out LAFF-.xM -> Programming -> Week -> QR unb.m This should equal, approximately, the zero matrix. Check if Q has mutually orthogonal columns: Q_out * Q_out

29 .4. Change of Basis 49.4 Change of Basis.4. The Unit Basis Vectors, One More Time Once again, recall the unit basis vectors in R : e =, e =. Now, 4 = 4 + by which we illustrate the fact that and form a basis for R and the vector 4 can then be written as a linear combination of these basis vectors, with coefficients 4 and. We can illustrate this with 4 = Change of Basis Similar to the example from the last unit, we could have created an alternate coordinate system with basis vectors q = =, q = =. What are the coefficients for the linear combination of these two vectors q and q that produce the vector 4? First let s look at a few exercises demonstrating how special these vectors that we ve chosen are.

30 Week. Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases 4 Homework.4.. The vectors q = =, q = =. are mutually orthonormal. True/False Homework.4.. If Q R n n has mutually orthonormal columns then which of the following are true:. Q T Q = I True/False. QQ T = I True/False 3. QQ = I True/False 4. Q = Q T True/False 4 What we would like to determine are the coefficients χ and χ such that χ + χ = 4. This can be alternatively written as Q χ χ = 4 In Homework.4.. we noticed that } {{ } Q T Q =

31 .4. Change of Basis 4 and hence Q T Q I χ χ = } {{ } Q T 4 or, equivalently, χ χ = } {{ } Q T 4 = = 3 so that 3 = 4. In other words: In the new basis, the coefficients are 3 and. Another way of thinking of the above discussion is that 4 + = = = 4 = 4 Q Q 3 Q T 4 = 3 = Q This last way of looking at the problem suggest a way of finding the coefficients for any basis, a,a,...,a n R n. Let b R n and let A = a a a n. Then b = AA I b = Ax = χ a + χ a + + χ n a n. So, when the basis is changed from the unit basis vectors to the vectors a,a,...,a n, the coefficients change from β,β,...,β n the components of the vector b to χ,χ,...,χ n the components of the vector x. Obviously, instead of computing A b, one can instead solve Ax = b.

32 Week. Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases 4.5 Singular Value Decomposition.5. The Best Low Rank Approximation Earlier this week, we showed that by taking a few columns from matrix B which encoded the picture, and projecting onto those columns we could create a rank-k approximation, AW T, that approximated the picture. The columns in A were chosen from the columns of B. Now, what if we could choose the columns of A to be the best colums onto which to project? In other words, what if we could choose the columns of A so that the subspace spanned by them minimized the error in the approximation AW T when we choose W = A T A A T B? The answer to how to obtain the answers the above questions go beyond the scope of an introductory undergraduate linear algebra course. But let us at least look at some of the results. One of the most important results in linear algebra is the Singular Value Decomposition Theorem which says that any matrix B R m n can be written as the product of three matrices, the Singular Value Decomposition SVD: where U R m r and U T U = I U has orthonormal columns. B = UΣV T Σ R r r is a diagonal matrix with positive diagonal elements that are ordered so that σ, σ, σ r,r >. V R n r and V T V = I V has orthonormal columns. r equals the rank of matrix B. If we partition U = U L U R,V = V L V R, and Σ = Σ T L Σ BR, where U L and V L have k columns and Σ T L is k k, then U L Σ T L VL T is the best rank-k approximation to matrix B. So, the best rank-k approximation B = AW T is given by the choices A = U L and W = Σ T L V L. The sequence of pictures in Figures. and.3 illustrate the benefits of using a rank-k update based on the SVD. Homework.5.. Let B = UΣV T be the SVD of B, with U R m r, Σ R r r, and V R n r. Partition σ σ U = u u u r, Σ =.....,V = v v v r.. σ r UΣV T = σ u v T + σ u v T + + σ r u r v T r. Always/Sometimes/Never Homework.5.. Let B = UΣV T be the SVD of B with U R m r, Σ R r r, and V R n r. CB = CU R B = CV Always/Sometimes/Never Always/Sometimes/Never

33 .5. Singular Value Decomposition 43 AA T A A T B U L Σ T L V T L k = k = k = 5 k = 5 k = k = Figure.: Rank-k approximation using columns from the picture versus using the SVD. Part

34 Week. Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases 44 AA T A A T B U L Σ T L V T L k = 5 k = 5 k = 5 k = 5 Figure.3: Rank-k approximation using columns from the picture versus using the SVD. Continued Given A R m n with linearly independent columns, and b R m, we can solve Ax b for the best solution in the linear least-squares sense via its SVD, A = UΣV T, by observing that ˆx = A T A A T b = UΣV T T UΣV T UΣV T T b = V Σ T U T UΣV T V Σ T U T b = V ΣΣV T V ΣU T b = V T ΣΣ V V ΣU T b = V Σ Σ ΣU T b = V Σ U T b. Hence, the best solution is given by ˆx = V Σ U T b.

35 .6. Enrichment 45 Homework.5..3 You will now want to revisit exercise..5. and compare an approximation by projecting onto a few columns of the picture versus using the SVD to approximate. You can do so by executing the script Week/CompressPictureWithSVD.m that you downloaded in Week.zip. That script creates three figures: the first is the original picture. The second is the approximation as we discussed in Section..5. The third uses the SVD. Play with the script, changing variable k..6 Enrichment.6. The Problem with Computing the QR Factorization Modified Gram-Schmidt In theory, the Gram-Schmidt process, started with a set of linearly independent vectors, yields an orthonormal basis for the span of those vectors. In practice, due to round-off error, the process can result in a set of vectors that are far from mutually orhonormal. A minor modification of the Gram-Schmidt process, known as Modified Gram-Schmidt, partially fixes this. A more advanced treatment of Gram-Schmidt orthonalization, including the Modified Gram-Schmidt process, can be found in Robert s notes for his graduate class on Numerical Linear Algebra, available from Many linear algebra texts also treat this material..6. QR Factorization Via Householder Transformations Reflections If orthogonality is important, an alternative algorithm for computing the QR factorization is employed, based on Householder transformations reflections. This approach resembles LU factorization with Gauss transforms, except that at each step a reflection is used to zero elements below the current diagonal. QR factorization via Householder transformations is discussed in Robert s notes for his graduate class on Numerical Linear Algebra, available from Graduate level texts on numerical linear algebra usually treat this topic, as may some more advanced undergraduate texts..6.3 More on SVD The SVD is possibly the most important topic in linear algebra. A thorough treatment of the SVD can be found in Robert s notes for his graduate class on Numerical Linear Algebra, available from Graduate level texts on numerical linear algebra usually treat this topic, as may some more advanced undergraduate texts..7 Wrap Up.7. Homework No additional homework this week..7. Summary Projection Given a,b R m : Component of b in direction of a: Matrix that projects onto Span{a}: u = at b a T a a = aat a a T b. aa T a a T

36 Week. Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases 46 Component of b orthogonal to a: Matrix that projects onto Span{a} : w = b at b a T a a = b aat a a T b = I aa T a a T b. I aa T a a T Given A R m n with linearly independent columns and vector b R m : Component of b in CA: Matrix that projects onto CA: Component of b in CA = N A T : Matrix that projects onto CA = N A T : u = AA T A A T b. AA T A A T. w = b AA T A A T b = I AA T A A T b. I AA T A A T. Best rank-k approximation of B R m n using the column space of A R m k with linearly independent columns: Orthonormal vectors and spaces AA T A A T B = AV T, where V T = A T A A T B. Definition.3 Let q,q,...,q k R m. Then these vectors are mutually orthonormal if for all i, j < k : q T if i = j i q j = otherwise. Theorem.4 A matrix Q R m n has mutually orthonormal columns if and only if Q T Q = I. Given q,b R m, with q = q of length one: Component of b in direction of q: Matrix that projects onto Span{q}: Component of b orthogonal to q: Matrix that projects onto Span{q} : u = q T bq = qq T b. qq T w = b q T bq = I qq T b. I qq T Given matrix Q R m n with mutually orthonormal columns and vector b R m : Component of b in CQ: Matrix that projects onto CQ: Component of b in CQ = N Q: Matrix that projects onto CQ = N Q T : u = QQ T b. QQ T. w = b QQ T b = I QQ T b. I QQ T. Best rank-k approximation of B R m n using the column space of Q R m k with mutually orthonormal columns: QQ T B = QV T, where V T = Q T B.

37 .7. Wrap Up 47 Gram-Schmidt orthogonalization Starting with linearly independent vectors a,a,...,a n R m, the following algorithm computes the mutually orthonormal vectors q,q,...,q n R m such that Span{a,a,...,a n } = Span{q,q,...,q n }: for k =,...,n for p =,...,k ρ p,k := q T p a k endfor a k := a k for j =,...,k a k := a k ρ j,kq j endfor ρ k,k := a k q k := a k /ρ k,k endfor } } ρ,k ρ,k. ρ k,k q T a k q T = a k =. q T k a k a k = a k k j= ρ j,kq j = a k Normalize a k to be of length one. q T q T. q T k a k = q q q k q q q k T ak ρ,k ρ,k. ρ k,k The QR factorization Given A R m n with linearly independent columns, there exists a matrix Q R m n with mutually orthonormal columns and upper triangular matrix R R n n such that A = QR. If one partitions ρ, ρ, ρ,n then A = a a a n, Q = a a a n A = q q q n, and R = q q q n Q ρ, ρ,n ρ n,n ρ, ρ, ρ,n ρ, ρ,n ρ n,n and Gram-Schmidt orthogonalization the Gram-Schmidt process in the above algorithm computes the columns of Q and elements of R. R Solving the linear least-squares problem via the QR factorization Given A R m n with linearly independent columns, there exists a matrix Q R m n with mutually orthonormal columns and upper triangular matrix R R n n such that A = QR. The vector ˆx that is the best solution in the linear least-squares sense to Ax b is given by ˆx = A T A A T b as shown in Week computed by solving the normal equations A T Ax = A T b.

38 Week. Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases 48 ˆx = R Q T b computed by solving Rx = Q T b. An algorithm for computing the QR factorization presented in FLAME notation is given by Algorithm: [Q, R] := QRA, Q, R Partition A A L A R, Q where A L and Q L have columns, R T L is while na L < na do Repartition A L A R R T L R BL R T R R BR A a A, R r R r T ρ r T R r R Q L Q R, R R T L Q L Q R R BL R T R R BR Q q Q, r := Q T a a := a Q r ρ := a q = a /ρ Continue with A L A R R T L R BL endwhile R T R R BR A a A, R r R r T ρ r T R r R Q L Q R Q q Q, Singular Value Decomposition Any matrix B R m n can be written as the product of three matrices, the Singular Value Decomposition SVD: B = UΣV T where U R m r and U T U = I U has orthonormal columns. Σ R r r is a diagonal matrix with positive diagonal elements that are ordered so that σ, σ, σ r,r >. V R n r and V T V = I V has orthonormal columns. r equals the rank of matrix B. If we partition U = U L U R,V = V L V R, and Σ = Σ T L Σ BR,

39 .7. Wrap Up 49 where U L and V L have k columns and Σ T L is k k, then U L Σ T L VL T is the best rank-k approximation to matrix B. So, the best rank-k approximation B = AW T is given by the choices A = U L and W = Σ T L V L. Given A R m n with linearly independent columns, and b R m, the best solution to Ax b in the linear least-squares sense via its SVD, A = UΣV T, is given by ˆx = V Σ U T b.

40 Week. Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases 4

Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases

Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases Week Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases. Opening Remarks.. Low Rank Approximation 463 Week. Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases 464.. Outline..

More information

Vector Spaces, Orthogonality, and Linear Least Squares

Vector Spaces, Orthogonality, and Linear Least Squares Week Vector Spaces, Orthogonality, and Linear Least Squares. Opening Remarks.. Visualizing Planes, Lines, and Solutions Consider the following system of linear equations from the opener for Week 9: χ χ

More information

Notes on Solving Linear Least-Squares Problems

Notes on Solving Linear Least-Squares Problems Notes on Solving Linear Least-Squares Problems Robert A. van de Geijn The University of Texas at Austin Austin, TX 7871 October 1, 14 NOTE: I have not thoroughly proof-read these notes!!! 1 Motivation

More information

Matrix-Vector Operations

Matrix-Vector Operations Week3 Matrix-Vector Operations 31 Opening Remarks 311 Timmy Two Space View at edx Homework 3111 Click on the below link to open a browser window with the Timmy Two Space exercise This exercise was suggested

More information

Matrix-Vector Operations

Matrix-Vector Operations Week3 Matrix-Vector Operations 31 Opening Remarks 311 Timmy Two Space View at edx Homework 3111 Click on the below link to open a browser window with the Timmy Two Space exercise This exercise was suggested

More information

Orthogonality. 6.1 Orthogonal Vectors and Subspaces. Chapter 6

Orthogonality. 6.1 Orthogonal Vectors and Subspaces. Chapter 6 Chapter 6 Orthogonality 6.1 Orthogonal Vectors and Subspaces Recall that if nonzero vectors x, y R n are linearly independent then the subspace of all vectors αx + βy, α, β R (the space spanned by x and

More information

Eigenvalues, Eigenvectors, and Diagonalization

Eigenvalues, Eigenvectors, and Diagonalization Week12 Eigenvalues, Eigenvectors, and Diagonalization 12.1 Opening Remarks 12.1.1 Predicting the Weather, Again View at edx Let us revisit the example from Week 4, in which we had a simple model for predicting

More information

Eigenvalues, Eigenvectors, and Diagonalization

Eigenvalues, Eigenvectors, and Diagonalization Week12 Eigenvalues, Eigenvectors, and Diagonalization 12.1 Opening Remarks 12.1.1 Predicting the Weather, Again Let us revisit the example from Week 4, in which we had a simple model for predicting the

More information

Applied Mathematics 205. Unit II: Numerical Linear Algebra. Lecturer: Dr. David Knezevic

Applied Mathematics 205. Unit II: Numerical Linear Algebra. Lecturer: Dr. David Knezevic Applied Mathematics 205 Unit II: Numerical Linear Algebra Lecturer: Dr. David Knezevic Unit II: Numerical Linear Algebra Chapter II.3: QR Factorization, SVD 2 / 66 QR Factorization 3 / 66 QR Factorization

More information

Linear Algebra- Final Exam Review

Linear Algebra- Final Exam Review Linear Algebra- Final Exam Review. Let A be invertible. Show that, if v, v, v 3 are linearly independent vectors, so are Av, Av, Av 3. NOTE: It should be clear from your answer that you know the definition.

More information

Chapter 5 Orthogonality

Chapter 5 Orthogonality Matrix Methods for Computational Modeling and Data Analytics Virginia Tech Spring 08 Chapter 5 Orthogonality Mark Embree embree@vt.edu Ax=b version of February 08 We needonemoretoolfrom basic linear algebra

More information

Linear Algebra, part 3 QR and SVD

Linear Algebra, part 3 QR and SVD Linear Algebra, part 3 QR and SVD Anna-Karin Tornberg Mathematical Models, Analysis and Simulation Fall semester, 2012 Going back to least squares (Section 1.4 from Strang, now also see section 5.2). We

More information

Vector Spaces. 9.1 Opening Remarks. Week Solvable or not solvable, that s the question. View at edx. Consider the picture

Vector Spaces. 9.1 Opening Remarks. Week Solvable or not solvable, that s the question. View at edx. Consider the picture Week9 Vector Spaces 9. Opening Remarks 9.. Solvable or not solvable, that s the question Consider the picture (,) (,) p(χ) = γ + γ χ + γ χ (, ) depicting three points in R and a quadratic polynomial (polynomial

More information

(a) If A is a 3 by 4 matrix, what does this tell us about its nullspace? Solution: dim N(A) 1, since rank(a) 3. Ax =

(a) If A is a 3 by 4 matrix, what does this tell us about its nullspace? Solution: dim N(A) 1, since rank(a) 3. Ax = . (5 points) (a) If A is a 3 by 4 matrix, what does this tell us about its nullspace? dim N(A), since rank(a) 3. (b) If we also know that Ax = has no solution, what do we know about the rank of A? C(A)

More information

AM 205: lecture 8. Last time: Cholesky factorization, QR factorization Today: how to compute the QR factorization, the Singular Value Decomposition

AM 205: lecture 8. Last time: Cholesky factorization, QR factorization Today: how to compute the QR factorization, the Singular Value Decomposition AM 205: lecture 8 Last time: Cholesky factorization, QR factorization Today: how to compute the QR factorization, the Singular Value Decomposition QR Factorization A matrix A R m n, m n, can be factorized

More information

Math 520 Exam 2 Topic Outline Sections 1 3 (Xiao/Dumas/Liaw) Spring 2008

Math 520 Exam 2 Topic Outline Sections 1 3 (Xiao/Dumas/Liaw) Spring 2008 Math 520 Exam 2 Topic Outline Sections 1 3 (Xiao/Dumas/Liaw) Spring 2008 Exam 2 will be held on Tuesday, April 8, 7-8pm in 117 MacMillan What will be covered The exam will cover material from the lectures

More information

Linear Algebra, part 3. Going back to least squares. Mathematical Models, Analysis and Simulation = 0. a T 1 e. a T n e. Anna-Karin Tornberg

Linear Algebra, part 3. Going back to least squares. Mathematical Models, Analysis and Simulation = 0. a T 1 e. a T n e. Anna-Karin Tornberg Linear Algebra, part 3 Anna-Karin Tornberg Mathematical Models, Analysis and Simulation Fall semester, 2010 Going back to least squares (Sections 1.7 and 2.3 from Strang). We know from before: The vector

More information

[Disclaimer: This is not a complete list of everything you need to know, just some of the topics that gave people difficulty.]

[Disclaimer: This is not a complete list of everything you need to know, just some of the topics that gave people difficulty.] Math 43 Review Notes [Disclaimer: This is not a complete list of everything you need to know, just some of the topics that gave people difficulty Dot Product If v (v, v, v 3 and w (w, w, w 3, then the

More information

MODULE 7. where A is an m n real (or complex) matrix. 2) Let K(t, s) be a function of two variables which is continuous on the square [0, 1] [0, 1].

MODULE 7. where A is an m n real (or complex) matrix. 2) Let K(t, s) be a function of two variables which is continuous on the square [0, 1] [0, 1]. Topics: Linear operators MODULE 7 We are going to discuss functions = mappings = transformations = operators from one vector space V 1 into another vector space V 2. However, we shall restrict our sights

More information

Orthonormal Bases; Gram-Schmidt Process; QR-Decomposition

Orthonormal Bases; Gram-Schmidt Process; QR-Decomposition Orthonormal Bases; Gram-Schmidt Process; QR-Decomposition MATH 322, Linear Algebra I J. Robert Buchanan Department of Mathematics Spring 205 Motivation When working with an inner product space, the most

More information

MODULE 8 Topics: Null space, range, column space, row space and rank of a matrix

MODULE 8 Topics: Null space, range, column space, row space and rank of a matrix MODULE 8 Topics: Null space, range, column space, row space and rank of a matrix Definition: Let L : V 1 V 2 be a linear operator. The null space N (L) of L is the subspace of V 1 defined by N (L) = {x

More information

18.06SC Final Exam Solutions

18.06SC Final Exam Solutions 18.06SC Final Exam Solutions 1 (4+7=11 pts.) Suppose A is 3 by 4, and Ax = 0 has exactly 2 special solutions: 1 2 x 1 = 1 and x 2 = 1 1 0 0 1 (a) Remembering that A is 3 by 4, find its row reduced echelon

More information

Worksheet for Lecture 25 Section 6.4 Gram-Schmidt Process

Worksheet for Lecture 25 Section 6.4 Gram-Schmidt Process Worksheet for Lecture Name: Section.4 Gram-Schmidt Process Goal For a subspace W = Span{v,..., v n }, we want to find an orthonormal basis of W. Example Let W = Span{x, x } with x = and x =. Give an orthogonal

More information

18.06 Professor Johnson Quiz 1 October 3, 2007

18.06 Professor Johnson Quiz 1 October 3, 2007 18.6 Professor Johnson Quiz 1 October 3, 7 SOLUTIONS 1 3 pts.) A given circuit network directed graph) which has an m n incidence matrix A rows = edges, columns = nodes) and a conductance matrix C [diagonal

More information

DS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra.

DS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra. DS-GA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1

More information

MATH 22A: LINEAR ALGEBRA Chapter 4

MATH 22A: LINEAR ALGEBRA Chapter 4 MATH 22A: LINEAR ALGEBRA Chapter 4 Jesús De Loera, UC Davis November 30, 2012 Orthogonality and Least Squares Approximation QUESTION: Suppose Ax = b has no solution!! Then what to do? Can we find an Approximate

More information

Applied Linear Algebra in Geoscience Using MATLAB

Applied Linear Algebra in Geoscience Using MATLAB Applied Linear Algebra in Geoscience Using MATLAB Contents Getting Started Creating Arrays Mathematical Operations with Arrays Using Script Files and Managing Data Two-Dimensional Plots Programming in

More information

Lecture 6, Sci. Comp. for DPhil Students

Lecture 6, Sci. Comp. for DPhil Students Lecture 6, Sci. Comp. for DPhil Students Nick Trefethen, Thursday 1.11.18 Today II.3 QR factorization II.4 Computation of the QR factorization II.5 Linear least-squares Handouts Quiz 4 Householder s 4-page

More information

Chapter 3 - From Gaussian Elimination to LU Factorization

Chapter 3 - From Gaussian Elimination to LU Factorization Chapter 3 - From Gaussian Elimination to LU Factorization Maggie Myers Robert A. van de Geijn The University of Texas at Austin Practical Linear Algebra Fall 29 http://z.cs.utexas.edu/wiki/pla.wiki/ 1

More information

MATH 167: APPLIED LINEAR ALGEBRA Least-Squares

MATH 167: APPLIED LINEAR ALGEBRA Least-Squares MATH 167: APPLIED LINEAR ALGEBRA Least-Squares October 30, 2014 Least Squares We do a series of experiments, collecting data. We wish to see patterns!! We expect the output b to be a linear function of

More information

Numerical Linear Algebra Chap. 2: Least Squares Problems

Numerical Linear Algebra Chap. 2: Least Squares Problems Numerical Linear Algebra Chap. 2: Least Squares Problems Heinrich Voss voss@tu-harburg.de Hamburg University of Technology Institute of Numerical Simulation TUHH Heinrich Voss Numerical Linear Algebra

More information

Chapter 6: Orthogonality

Chapter 6: Orthogonality Chapter 6: Orthogonality (Last Updated: November 7, 7) These notes are derived primarily from Linear Algebra and its applications by David Lay (4ed). A few theorems have been moved around.. Inner products

More information

Stability of the Gram-Schmidt process

Stability of the Gram-Schmidt process Stability of the Gram-Schmidt process Orthogonal projection We learned in multivariable calculus (or physics or elementary linear algebra) that if q is a unit vector and v is any vector then the orthogonal

More information

Stat 159/259: Linear Algebra Notes

Stat 159/259: Linear Algebra Notes Stat 159/259: Linear Algebra Notes Jarrod Millman November 16, 2015 Abstract These notes assume you ve taken a semester of undergraduate linear algebra. In particular, I assume you are familiar with the

More information

LINEAR ALGEBRA KNOWLEDGE SURVEY

LINEAR ALGEBRA KNOWLEDGE SURVEY LINEAR ALGEBRA KNOWLEDGE SURVEY Instructions: This is a Knowledge Survey. For this assignment, I am only interested in your level of confidence about your ability to do the tasks on the following pages.

More information

Linear Algebra Primer

Linear Algebra Primer Linear Algebra Primer David Doria daviddoria@gmail.com Wednesday 3 rd December, 2008 Contents Why is it called Linear Algebra? 4 2 What is a Matrix? 4 2. Input and Output.....................................

More information

A Review of Linear Algebra

A Review of Linear Algebra A Review of Linear Algebra Mohammad Emtiyaz Khan CS,UBC A Review of Linear Algebra p.1/13 Basics Column vector x R n, Row vector x T, Matrix A R m n. Matrix Multiplication, (m n)(n k) m k, AB BA. Transpose

More information

. = V c = V [x]v (5.1) c 1. c k

. = V c = V [x]v (5.1) c 1. c k Chapter 5 Linear Algebra It can be argued that all of linear algebra can be understood using the four fundamental subspaces associated with a matrix Because they form the foundation on which we later work,

More information

33AH, WINTER 2018: STUDY GUIDE FOR FINAL EXAM

33AH, WINTER 2018: STUDY GUIDE FOR FINAL EXAM 33AH, WINTER 2018: STUDY GUIDE FOR FINAL EXAM (UPDATED MARCH 17, 2018) The final exam will be cumulative, with a bit more weight on more recent material. This outline covers the what we ve done since the

More information

Math 102, Winter Final Exam Review. Chapter 1. Matrices and Gaussian Elimination

Math 102, Winter Final Exam Review. Chapter 1. Matrices and Gaussian Elimination Math 0, Winter 07 Final Exam Review Chapter. Matrices and Gaussian Elimination { x + x =,. Different forms of a system of linear equations. Example: The x + 4x = 4. [ ] [ ] [ ] vector form (or the column

More information

From Matrix-Vector Multiplication to Matrix-Matrix Multiplication

From Matrix-Vector Multiplication to Matrix-Matrix Multiplication Week4 From Matrix-Vector Multiplication to Matrix-Matrix Multiplication here are a LO of programming assignments this week hey are meant to help clarify slicing and dicing hey show that the right abstractions

More information

Math 4A Notes. Written by Victoria Kala Last updated June 11, 2017

Math 4A Notes. Written by Victoria Kala Last updated June 11, 2017 Math 4A Notes Written by Victoria Kala vtkala@math.ucsb.edu Last updated June 11, 2017 Systems of Linear Equations A linear equation is an equation that can be written in the form a 1 x 1 + a 2 x 2 +...

More information

Review problems for MA 54, Fall 2004.

Review problems for MA 54, Fall 2004. Review problems for MA 54, Fall 2004. Below are the review problems for the final. They are mostly homework problems, or very similar. If you are comfortable doing these problems, you should be fine on

More information

orthogonal relations between vectors and subspaces Then we study some applications in vector spaces and linear systems, including Orthonormal Basis,

orthogonal relations between vectors and subspaces Then we study some applications in vector spaces and linear systems, including Orthonormal Basis, 5 Orthogonality Goals: We use scalar products to find the length of a vector, the angle between 2 vectors, projections, orthogonal relations between vectors and subspaces Then we study some applications

More information

Householder reflectors are matrices of the form. P = I 2ww T, where w is a unit vector (a vector of 2-norm unity)

Householder reflectors are matrices of the form. P = I 2ww T, where w is a unit vector (a vector of 2-norm unity) Householder QR Householder reflectors are matrices of the form P = I 2ww T, where w is a unit vector (a vector of 2-norm unity) w Px x Geometrically, P x represents a mirror image of x with respect to

More information

7. Dimension and Structure.

7. Dimension and Structure. 7. Dimension and Structure 7.1. Basis and Dimension Bases for Subspaces Example 2 The standard unit vectors e 1, e 2,, e n are linearly independent, for if we write (2) in component form, then we obtain

More information

Numerical Linear Algebra

Numerical Linear Algebra Numerical Linear Algebra The two principal problems in linear algebra are: Linear system Given an n n matrix A and an n-vector b, determine x IR n such that A x = b Eigenvalue problem Given an n n matrix

More information

Linear Algebra, Summer 2011, pt. 2

Linear Algebra, Summer 2011, pt. 2 Linear Algebra, Summer 2, pt. 2 June 8, 2 Contents Inverses. 2 Vector Spaces. 3 2. Examples of vector spaces..................... 3 2.2 The column space......................... 6 2.3 The null space...........................

More information

Linear Algebra Fundamentals

Linear Algebra Fundamentals Linear Algebra Fundamentals It can be argued that all of linear algebra can be understood using the four fundamental subspaces associated with a matrix. Because they form the foundation on which we later

More information

STAT 309: MATHEMATICAL COMPUTATIONS I FALL 2018 LECTURE 9

STAT 309: MATHEMATICAL COMPUTATIONS I FALL 2018 LECTURE 9 STAT 309: MATHEMATICAL COMPUTATIONS I FALL 2018 LECTURE 9 1. qr and complete orthogonal factorization poor man s svd can solve many problems on the svd list using either of these factorizations but they

More information

Singular Value Decomposition

Singular Value Decomposition Singular Value Decomposition Motivatation The diagonalization theorem play a part in many interesting applications. Unfortunately not all matrices can be factored as A = PDP However a factorization A =

More information

Orthogonality. Orthonormal Bases, Orthogonal Matrices. Orthogonality

Orthogonality. Orthonormal Bases, Orthogonal Matrices. Orthogonality Orthonormal Bases, Orthogonal Matrices The Major Ideas from Last Lecture Vector Span Subspace Basis Vectors Coordinates in different bases Matrix Factorization (Basics) The Major Ideas from Last Lecture

More information

B553 Lecture 5: Matrix Algebra Review

B553 Lecture 5: Matrix Algebra Review B553 Lecture 5: Matrix Algebra Review Kris Hauser January 19, 2012 We have seen in prior lectures how vectors represent points in R n and gradients of functions. Matrices represent linear transformations

More information

Basic Elements of Linear Algebra

Basic Elements of Linear Algebra A Basic Review of Linear Algebra Nick West nickwest@stanfordedu September 16, 2010 Part I Basic Elements of Linear Algebra Although the subject of linear algebra is much broader than just vectors and matrices,

More information

Chapter 6 - Orthogonality

Chapter 6 - Orthogonality Chapter 6 - Orthogonality Maggie Myers Robert A. van de Geijn The University of Texas at Austin Orthogonality Fall 2009 http://z.cs.utexas.edu/wiki/pla.wiki/ 1 Orthogonal Vectors and Subspaces http://z.cs.utexas.edu/wiki/pla.wiki/

More information

(v, w) = arccos( < v, w >

(v, w) = arccos( < v, w > MA322 Sathaye Notes on Inner Products Notes on Chapter 6 Inner product. Given a real vector space V, an inner product is defined to be a bilinear map F : V V R such that the following holds: For all v

More information

Lecture notes: Applied linear algebra Part 1. Version 2

Lecture notes: Applied linear algebra Part 1. Version 2 Lecture notes: Applied linear algebra Part 1. Version 2 Michael Karow Berlin University of Technology karow@math.tu-berlin.de October 2, 2008 1 Notation, basic notions and facts 1.1 Subspaces, range and

More information

COMP 558 lecture 18 Nov. 15, 2010

COMP 558 lecture 18 Nov. 15, 2010 Least squares We have seen several least squares problems thus far, and we will see more in the upcoming lectures. For this reason it is good to have a more general picture of these problems and how to

More information

Notes on Householder QR Factorization

Notes on Householder QR Factorization Notes on Householder QR Factorization Robert A van de Geijn Department of Computer Science he University of exas at Austin Austin, X 7872 rvdg@csutexasedu September 2, 24 Motivation A fundamental problem

More information

5.) For each of the given sets of vectors, determine whether or not the set spans R 3. Give reasons for your answers.

5.) For each of the given sets of vectors, determine whether or not the set spans R 3. Give reasons for your answers. Linear Algebra - Test File - Spring Test # For problems - consider the following system of equations. x + y - z = x + y + 4z = x + y + 6z =.) Solve the system without using your calculator..) Find the

More information

Assignment #9: Orthogonal Projections, Gram-Schmidt, and Least Squares. Name:

Assignment #9: Orthogonal Projections, Gram-Schmidt, and Least Squares. Name: Assignment 9: Orthogonal Projections, Gram-Schmidt, and Least Squares Due date: Friday, April 0, 08 (:pm) Name: Section Number Assignment 9: Orthogonal Projections, Gram-Schmidt, and Least Squares Due

More information

Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012

Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012 Instructions Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012 The exam consists of four problems, each having multiple parts. You should attempt to solve all four problems. 1.

More information

Matrix-Matrix Multiplication

Matrix-Matrix Multiplication Week5 Matrix-Matrix Multiplication 51 Opening Remarks 511 Composing Rotations Homework 5111 Which of the following statements are true: cosρ + σ + τ cosτ sinτ cosρ + σ sinρ + σ + τ sinτ cosτ sinρ + σ cosρ

More information

AMS526: Numerical Analysis I (Numerical Linear Algebra)

AMS526: Numerical Analysis I (Numerical Linear Algebra) AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 5: Projectors and QR Factorization Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 14 Outline 1 Projectors 2 QR Factorization

More information

Week6. Gaussian Elimination. 6.1 Opening Remarks Solving Linear Systems. View at edx

Week6. Gaussian Elimination. 6.1 Opening Remarks Solving Linear Systems. View at edx Week6 Gaussian Elimination 61 Opening Remarks 611 Solving Linear Systems View at edx 193 Week 6 Gaussian Elimination 194 61 Outline 61 Opening Remarks 193 611 Solving Linear Systems 193 61 Outline 194

More information

A Brief Outline of Math 355

A Brief Outline of Math 355 A Brief Outline of Math 355 Lecture 1 The geometry of linear equations; elimination with matrices A system of m linear equations with n unknowns can be thought of geometrically as m hyperplanes intersecting

More information

Linear Models Review

Linear Models Review Linear Models Review Vectors in IR n will be written as ordered n-tuples which are understood to be column vectors, or n 1 matrices. A vector variable will be indicted with bold face, and the prime sign

More information

MAT2342 : Introduction to Applied Linear Algebra Mike Newman, fall Projections. introduction

MAT2342 : Introduction to Applied Linear Algebra Mike Newman, fall Projections. introduction MAT4 : Introduction to Applied Linear Algebra Mike Newman fall 7 9. Projections introduction One reason to consider projections is to understand approximate solutions to linear systems. A common example

More information

Department of Aerospace Engineering AE602 Mathematics for Aerospace Engineers Assignment No. 4

Department of Aerospace Engineering AE602 Mathematics for Aerospace Engineers Assignment No. 4 Department of Aerospace Engineering AE6 Mathematics for Aerospace Engineers Assignment No.. Decide whether or not the following vectors are linearly independent, by solving c v + c v + c 3 v 3 + c v :

More information

AMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences)

AMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences) AMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences) Lecture 1: Course Overview; Matrix Multiplication Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical

More information

(v, w) = arccos( < v, w >

(v, w) = arccos( < v, w > MA322 F all206 Notes on Inner Products Notes on Chapter 6 Inner product. Given a real vector space V, an inner product is defined to be a bilinear map F : V V R such that the following holds: Commutativity:

More information

BASIC NOTIONS. x + y = 1 3, 3x 5y + z = A + 3B,C + 2D, DC are not defined. A + C =

BASIC NOTIONS. x + y = 1 3, 3x 5y + z = A + 3B,C + 2D, DC are not defined. A + C = CHAPTER I BASIC NOTIONS (a) 8666 and 8833 (b) a =6,a =4 will work in the first case, but there are no possible such weightings to produce the second case, since Student and Student 3 have to end up with

More information

Maths for Signals and Systems Linear Algebra in Engineering

Maths for Signals and Systems Linear Algebra in Engineering Maths for Signals and Systems Linear Algebra in Engineering Lecture 18, Friday 18 th November 2016 DR TANIA STATHAKI READER (ASSOCIATE PROFFESOR) IN SIGNAL PROCESSING IMPERIAL COLLEGE LONDON Mathematics

More information

Dot product and linear least squares problems

Dot product and linear least squares problems Dot product and linear least squares problems Dot Product For vectors u,v R n we define the dot product Note that we can also write this as u v = u,,u n u v = u v + + u n v n v v n = u v + + u n v n The

More information

(v, w) = arccos( < v, w >

(v, w) = arccos( < v, w > MA322 F all203 Notes on Inner Products Notes on Chapter 6 Inner product. Given a real vector space V, an inner product is defined to be a bilinear map F : V V R such that the following holds: For all v,

More information

EE731 Lecture Notes: Matrix Computations for Signal Processing

EE731 Lecture Notes: Matrix Computations for Signal Processing EE731 Lecture Notes: Matrix Computations for Signal Processing James P. Reilly c Department of Electrical and Computer Engineering McMaster University October 17, 005 Lecture 3 3 he Singular Value Decomposition

More information

EE731 Lecture Notes: Matrix Computations for Signal Processing

EE731 Lecture Notes: Matrix Computations for Signal Processing EE731 Lecture Notes: Matrix Computations for Signal Processing James P. Reilly c Department of Electrical and Computer Engineering McMaster University September 22, 2005 0 Preface This collection of ten

More information

Least squares: the big idea

Least squares: the big idea Notes for 2016-02-22 Least squares: the big idea Least squares problems are a special sort of minimization problem. Suppose A R m n where m > n. In general, we cannot solve the overdetermined system Ax

More information

Chapter 7: Symmetric Matrices and Quadratic Forms

Chapter 7: Symmetric Matrices and Quadratic Forms Chapter 7: Symmetric Matrices and Quadratic Forms (Last Updated: December, 06) These notes are derived primarily from Linear Algebra and its applications by David Lay (4ed). A few theorems have been moved

More information

Problem Set (T) If A is an m n matrix, B is an n p matrix and D is a p s matrix, then show

Problem Set (T) If A is an m n matrix, B is an n p matrix and D is a p s matrix, then show MTH 0: Linear Algebra Department of Mathematics and Statistics Indian Institute of Technology - Kanpur Problem Set Problems marked (T) are for discussions in Tutorial sessions (T) If A is an m n matrix,

More information

OHSx XM511 Linear Algebra: Solutions to Online True/False Exercises

OHSx XM511 Linear Algebra: Solutions to Online True/False Exercises This document gives the solutions to all of the online exercises for OHSx XM511. The section ( ) numbers refer to the textbook. TYPE I are True/False. Answers are in square brackets [. Lecture 02 ( 1.1)

More information

More Gaussian Elimination and Matrix Inversion

More Gaussian Elimination and Matrix Inversion Week7 More Gaussian Elimination and Matrix Inversion 7 Opening Remarks 7 Introduction 235 Week 7 More Gaussian Elimination and Matrix Inversion 236 72 Outline 7 Opening Remarks 235 7 Introduction 235 72

More information

Final Review Sheet. B = (1, 1 + 3x, 1 + x 2 ) then 2 + 3x + 6x 2

Final Review Sheet. B = (1, 1 + 3x, 1 + x 2 ) then 2 + 3x + 6x 2 Final Review Sheet The final will cover Sections Chapters 1,2,3 and 4, as well as sections 5.1-5.4, 6.1-6.2 and 7.1-7.3 from chapters 5,6 and 7. This is essentially all material covered this term. Watch

More information

MATH 221: SOLUTIONS TO SELECTED HOMEWORK PROBLEMS

MATH 221: SOLUTIONS TO SELECTED HOMEWORK PROBLEMS MATH 221: SOLUTIONS TO SELECTED HOMEWORK PROBLEMS 1. HW 1: Due September 4 1.1.21. Suppose v, w R n and c is a scalar. Prove that Span(v + cw, w) = Span(v, w). We must prove two things: that every element

More information

Practical Linear Algebra: A Geometry Toolbox

Practical Linear Algebra: A Geometry Toolbox Practical Linear Algebra: A Geometry Toolbox Third edition Chapter 12: Gauss for Linear Systems Gerald Farin & Dianne Hansford CRC Press, Taylor & Francis Group, An A K Peters Book www.farinhansford.com/books/pla

More information

Least Squares. Tom Lyche. October 26, Centre of Mathematics for Applications, Department of Informatics, University of Oslo

Least Squares. Tom Lyche. October 26, Centre of Mathematics for Applications, Department of Informatics, University of Oslo Least Squares Tom Lyche Centre of Mathematics for Applications, Department of Informatics, University of Oslo October 26, 2010 Linear system Linear system Ax = b, A C m,n, b C m, x C n. under-determined

More information

ENGG5781 Matrix Analysis and Computations Lecture 8: QR Decomposition

ENGG5781 Matrix Analysis and Computations Lecture 8: QR Decomposition ENGG5781 Matrix Analysis and Computations Lecture 8: QR Decomposition Wing-Kin (Ken) Ma 2017 2018 Term 2 Department of Electronic Engineering The Chinese University of Hong Kong Lecture 8: QR Decomposition

More information

Dimension and Structure

Dimension and Structure 96 Chapter 7 Dimension and Structure 7.1 Basis and Dimensions Bases for Subspaces Definition 7.1.1. A set of vectors in a subspace V of R n is said to be a basis for V if it is linearly independent and

More information

The Singular Value Decomposition and Least Squares Problems

The Singular Value Decomposition and Least Squares Problems The Singular Value Decomposition and Least Squares Problems Tom Lyche Centre of Mathematics for Applications, Department of Informatics, University of Oslo September 27, 2009 Applications of SVD solving

More information

4.1 Distance and Length

4.1 Distance and Length Chapter Vector Geometry In this chapter we will look more closely at certain geometric aspects of vectors in R n. We will first develop an intuitive understanding of some basic concepts by looking at vectors

More information

Vector Geometry. Chapter 5

Vector Geometry. Chapter 5 Chapter 5 Vector Geometry In this chapter we will look more closely at certain geometric aspects of vectors in R n. We will first develop an intuitive understanding of some basic concepts by looking at

More information

I. Multiple Choice Questions (Answer any eight)

I. Multiple Choice Questions (Answer any eight) Name of the student : Roll No : CS65: Linear Algebra and Random Processes Exam - Course Instructor : Prashanth L.A. Date : Sep-24, 27 Duration : 5 minutes INSTRUCTIONS: The test will be evaluated ONLY

More information

The value of a problem is not so much coming up with the answer as in the ideas and attempted ideas it forces on the would be solver I.N.

The value of a problem is not so much coming up with the answer as in the ideas and attempted ideas it forces on the would be solver I.N. Math 410 Homework Problems In the following pages you will find all of the homework problems for the semester. Homework should be written out neatly and stapled and turned in at the beginning of class

More information

There are six more problems on the next two pages

There are six more problems on the next two pages Math 435 bg & bu: Topics in linear algebra Summer 25 Final exam Wed., 8/3/5. Justify all your work to receive full credit. Name:. Let A 3 2 5 Find a permutation matrix P, a lower triangular matrix L with

More information

Linear Algebra. Session 12

Linear Algebra. Session 12 Linear Algebra. Session 12 Dr. Marco A Roque Sol 08/01/2017 Example 12.1 Find the constant function that is the least squares fit to the following data x 0 1 2 3 f(x) 1 0 1 2 Solution c = 1 c = 0 f (x)

More information

Lecture 3: Linear Algebra Review, Part II

Lecture 3: Linear Algebra Review, Part II Lecture 3: Linear Algebra Review, Part II Brian Borchers January 4, Linear Independence Definition The vectors v, v,..., v n are linearly independent if the system of equations c v + c v +...+ c n v n

More information

18.06 Problem Set 10 - Solutions Due Thursday, 29 November 2007 at 4 pm in

18.06 Problem Set 10 - Solutions Due Thursday, 29 November 2007 at 4 pm in 86 Problem Set - Solutions Due Thursday, 29 November 27 at 4 pm in 2-6 Problem : (5=5+5+5) Take any matrix A of the form A = B H CB, where B has full column rank and C is Hermitian and positive-definite

More information

1. Background: The SVD and the best basis (questions selected from Ch. 6- Can you fill in the exercises?)

1. Background: The SVD and the best basis (questions selected from Ch. 6- Can you fill in the exercises?) Math 35 Exam Review SOLUTIONS Overview In this third of the course we focused on linear learning algorithms to model data. summarize: To. Background: The SVD and the best basis (questions selected from

More information

Linear Transformations and Matrices

Linear Transformations and Matrices Week2 Linear Transformations and Matrices 2 Opening Remarks 2 Rotating in 2D View at edx Let R θ : R 2 R 2 be the function that rotates an input vector through an angle θ: R θ (x) θ x Figure 2 illustrates

More information

LINEAR ALGEBRA 1, 2012-I PARTIAL EXAM 3 SOLUTIONS TO PRACTICE PROBLEMS

LINEAR ALGEBRA 1, 2012-I PARTIAL EXAM 3 SOLUTIONS TO PRACTICE PROBLEMS LINEAR ALGEBRA, -I PARTIAL EXAM SOLUTIONS TO PRACTICE PROBLEMS Problem (a) For each of the two matrices below, (i) determine whether it is diagonalizable, (ii) determine whether it is orthogonally diagonalizable,

More information