AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 5: Projectors and QR Factorization Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 14
Outline 1 Projectors 2 QR Factorization Xiangmin Jiao Numerical Analysis I 2 / 14
Projectors A projector satisfies P 2 = P. They are also said to be idempotent. Orthogonal projector. Oblique projector. [ ] 0 0 Example: α 1 is an oblique projector if α 0, is orthogonal projector if α = 0. Xiangmin Jiao Numerical Analysis I 3 / 14
Projectors A projector satisfies P 2 = P. They are also said to be idempotent. Orthogonal projector. Oblique projector. [ ] 0 0 Example: α 1 is an oblique projector if α 0, is orthogonal projector if α = 0. Xiangmin Jiao Numerical Analysis I 3 / 14
Complementary Projectors Complementary projectors: P vs. I P. What space does I P project? Xiangmin Jiao Numerical Analysis I 4 / 14
Complementary Projectors Complementary projectors: P vs. I P. What space does I P project? Answer: null(p). range(i P) null(p) because Pv = 0 (I P)v = v. range(i P) null(p) because for any v (I P)v = v Pv null(p). A projector separates C m into two complementary subspace: range space and null space (i.e., range(p) + null(p) = C m and range(p) null(p) = {0} for projector P C m m ) It projects onto range space along null space In other words, x = Px + r, where r null(p) Question: Are range space and null space of projector orthogonal to each other? Xiangmin Jiao Numerical Analysis I 4 / 14
Orthogonal Projector An orthogonal projector is one that projects onto a subspace S 1 along a space S 2, where S 1 and S 2 are orthogonal. Theorem A projector P is orthogonal if and only if P = P. Xiangmin Jiao Numerical Analysis I 5 / 14
Orthogonal Projector An orthogonal projector is one that projects onto a subspace S 1 along a space S 2, where S 1 and S 2 are orthogonal. Theorem A projector P is orthogonal if and only if P = P. Proof. If direction: If P = P, then (Px) (I P)y = x (P P 2 )y = 0. Only if direction: Use SVD. Suppose P projects onto S 1 along S 2 where S 1 S 2, and S 1 has dimension n. Let {q 1,...,, q n } be orthonormal basis of S 1 and {q n+1,..., q m } be a basis for S 2. Let Q be unitary matrix whose jth column is q j, and we have PQ = (q 1, q 2,..., q n, 0,..., 0), so Q PQ = diag(1, 1,, 1, 0, ) = Σ, and P = QΣQ. Xiangmin Jiao Numerical Analysis I 5 / 14
Orthogonal Projector An orthogonal projector is one that projects onto a subspace S 1 along a space S 2, where S 1 and S 2 are orthogonal. Theorem A projector P is orthogonal if and only if P = P. Proof. If direction: If P = P, then (Px) (I P)y = x (P P 2 )y = 0. Only if direction: Use SVD. Suppose P projects onto S 1 along S 2 where S 1 S 2, and S 1 has dimension n. Let {q 1,...,, q n } be orthonormal basis of S 1 and {q n+1,..., q m } be a basis for S 2. Let Q be unitary matrix whose jth column is q j, and we have PQ = (q 1, q 2,..., q n, 0,..., 0), so Q PQ = diag(1, 1,, 1, 0, ) = Σ, and P = QΣQ. Question: Are orthogonal projectors orthogonal matrices? Xiangmin Jiao Numerical Analysis I 5 / 14
Basis of Projections Projection with orthonormal basis Given any matrix ˆQ C m n whose columns are orthonormal, then P = ˆQ ˆQ is orthogonal projector, so is I P We write I P as P In particular, if ˆQ = q, we write Pq = qq and P q = I Pq For arbitrary vector a, we write Pa = a aa a and P a = I Pa Xiangmin Jiao Numerical Analysis I 6 / 14
Basis of Projections Projection with orthonormal basis Given any matrix ˆQ C m n whose columns are orthonormal, then P = ˆQ ˆQ is orthogonal projector, so is I P We write I P as P In particular, if ˆQ = q, we write Pq = qq and P q = I Pq For arbitrary vector a, we write Pa = a aa a and P a = I Pa Projection with arbitrary basis Given any matrix A C m n that has full rank and m n P = A(A A) 1 A is orthogonal projection What does P project onto? Xiangmin Jiao Numerical Analysis I 6 / 14
Basis of Projections Projection with orthonormal basis Given any matrix ˆQ C m n whose columns are orthonormal, then P = ˆQ ˆQ is orthogonal projector, so is I P We write I P as P In particular, if ˆQ = q, we write Pq = qq and P q = I Pq For arbitrary vector a, we write Pa = a aa a and P a = I Pa Projection with arbitrary basis Given any matrix A C m n that has full rank and m n is orthogonal projection What does P project onto? range(a) P = A(A A) 1 A (A A) 1 A is called the pseudo-inverse of A, denoted as A + Xiangmin Jiao Numerical Analysis I 6 / 14
Outline 1 Projectors 2 QR Factorization Xiangmin Jiao Numerical Analysis I 7 / 14
Motivation Question: Given a linear system Ax b where A C m n (m n) has full rank, how to solve the linear system? Xiangmin Jiao Numerical Analysis I 8 / 14
Motivation Question: Given a linear system Ax b where A C m n (m n) has full rank, how to solve the linear system? Answer: One possible solution is to use SVD. How? Xiangmin Jiao Numerical Analysis I 8 / 14
Motivation Question: Given a linear system Ax b where A C m n (m n) has full rank, how to solve the linear system? Answer: One possible solution is to use SVD. How? A = UΣV, so x = V Σ 1 U b. Xiangmin Jiao Numerical Analysis I 8 / 14
Motivation Question: Given a linear system Ax b where A C m n (m n) has full rank, how to solve the linear system? Answer: One possible solution is to use SVD. How? A = UΣV, so x = V Σ 1 U b. Another solution is to use QR factorization, which decompose A into product of two simple matrices Q and R where columns of Q are orthonormal and R is upper triangular. Xiangmin Jiao Numerical Analysis I 8 / 14
Two Different Versions of QR Analogous to SVD, there are two versions of QR Full QR factorization: A C m n (m n) A = QR where Q C m m is unitary and R C m n is upper triangular Reduced QR factorization: A C m n (m n) A = Q R where Q C m n contains orthonormal vectors and R C n n is upper triangular What space do {q 1, q 2,, q j }, j n span? Xiangmin Jiao Numerical Analysis I 9 / 14
Two Different Versions of QR Analogous to SVD, there are two versions of QR Full QR factorization: A C m n (m n) A = QR where Q C m m is unitary and R C m n is upper triangular Reduced QR factorization: A C m n (m n) A = Q R where Q C m n contains orthonormal vectors and R C n n is upper triangular What space do {q 1, q 2,, q j }, j n span? Answer: For full rank A, first j column vectors of A, i.e., q 1, q 2,..., q j = a 1, a 2,..., a j. Xiangmin Jiao Numerical Analysis I 9 / 14
Gram-Schmidt Orthogonalization A method to construct QR factorization is to orthogonalize the column vectors of A: Basic idea: Take first column a 1 and normalize it to obtain vector q 1 ; Take second column a2, subtract its orthogonal projection to q 1, and normalize to obtain q 2 ;... Take jth column of a j, subtract its orthogonal projection to q 1,..., q j 1, and normalize to obtain q j ; j 1 v j = a j q i a j q i, q j = v j / v j. i=1 This idea is called Gram-Schmidt orthogonalization. Xiangmin Jiao Numerical Analysis I 10 / 14
Gram-Schmidt Projections Orthogonal vectors produced by Gram-Schmidt can be written in terms of projectors q j = P ja j P j a j where P j = I ˆQ j 1 ˆQ j 1 with ˆQ j 1 = [ q 1 q 2 q j 1 ] P j projects orthogonally onto space orthogonal to q 1, q 2,..., q j 1 and rank of P j is m (j 1) Xiangmin Jiao Numerical Analysis I 11 / 14
Algorithm of Gram-Schmidt Orthogonalization Classical Gram-Schmidt method for j = 1 to n v j = a j for i = 1 to j 1 r ij = q i a j v j = v j r ij q i r jj = v j 2 q j = v j /r jj Xiangmin Jiao Numerical Analysis I 12 / 14
Algorithm of Gram-Schmidt Orthogonalization Classical Gram-Schmidt method for j = 1 to n v j = a j for i = 1 to j 1 r ij = q i a j v j = v j r ij q i r jj = v j 2 q j = v j /r jj Classical Gram-Schmidt (CGS) is unstable, which means that its solution is sensitive to perturbation Xiangmin Jiao Numerical Analysis I 12 / 14
Existence of QR Theorem Every A C m n (m n) has full QR factorization, hence also a reduced QR factorization. Xiangmin Jiao Numerical Analysis I 13 / 14
Existence of QR Theorem Every A C m n (m n) has full QR factorization, hence also a reduced QR factorization. Key idea of proof: If A has full rank, Gram-Schmidt algorithm provides a proof itself for having reduced QR. If A does not have full rank, at some step v j = 0. We can set q j to be a vector orthogonal to q i, i < j. To construct full QR from reduced QR, just continue Gram-Schmidt an additional m n steps. Xiangmin Jiao Numerical Analysis I 13 / 14
Uniqueness of QR Theorem Every A C m n (m n) of full rank has a unique reduced QR factorization A = Q R with r jj > 0. Key idea of proof: Proof is provided by Gram-Schmidt iteration itself. If the signs of r jj are determined, then r ij and q j are determined. Xiangmin Jiao Numerical Analysis I 14 / 14
Uniqueness of QR Theorem Every A C m n (m n) of full rank has a unique reduced QR factorization A = Q R with r jj > 0. Key idea of proof: Proof is provided by Gram-Schmidt iteration itself. If the signs of r jj are determined, then r ij and q j are determined. Question: Why do we require r jj > 0? Xiangmin Jiao Numerical Analysis I 14 / 14
Uniqueness of QR Theorem Every A C m n (m n) of full rank has a unique reduced QR factorization A = Q R with r jj > 0. Key idea of proof: Proof is provided by Gram-Schmidt iteration itself. If the signs of r jj are determined, then r ij and q j are determined. Question: Why do we require r jj > 0? Question: Is full QR factorization unique? Xiangmin Jiao Numerical Analysis I 14 / 14
Uniqueness of QR Theorem Every A C m n (m n) of full rank has a unique reduced QR factorization A = Q R with r jj > 0. Key idea of proof: Proof is provided by Gram-Schmidt iteration itself. If the signs of r jj are determined, then r ij and q j are determined. Question: Why do we require r jj > 0? Question: Is full QR factorization unique? Question: What if A does not have full rank? Xiangmin Jiao Numerical Analysis I 14 / 14