1. The Polar Decomposition

Size: px

Start display at page:

Download "1. The Polar Decomposition"

Gilbert Black
6 years ago
Views:

1 A PERSONAL INTERVIEW WITH THE SINGULAR VALUE DECOMPOSITION MATAN GAVISH Part. Theory. The Polar Decomposition In what follows, F denotes either R or C. The vector space F n is an inner product space with the standard inner product,,. Let us denote by M n m (F) the set of matrices over F with n rows and m columns. Vectors will denote columns, that is, we will write F n v = v. The following fact is known as the (Left) Polar Decomposition: Let m n. Any matrix A M n m (F) may be factorized as A = U P where U M n m (F) has a orthonormal columns and P M m m (F) is positive semi-denite. Remark. When F = R and n = m, the polar decomposition makes precise the statement, that every linear transformation T : R n R n is a composition of a dilation and a rotation/reection. The proof will imply that in this case there is an orthonormal base (the eigenvectors of the matrix A A) on which A acts as a rotation/reection (U) composed with a dilation (P ). The polar decomposition follows directly from a remarkable connection between the (general) matrix A and the positive semi-denite matrix P = A A. Date: June 30, 200.

2 A PERSONAL INTERVIEW WITH THE SINGULAR VALUE DECOMPOSITION 2.. The Matrix A A. We rst note that the matrix A A M m m (F) is self-adjoint and positive semi-denite, since for any v F m we have 0 Av 2 = Av, Av = v, A Av = v A Av. There are several equivalent ways to dene the matrix A A. To name one, note that A A can be diagonalized over F, A A = W DW, where W is unitary and D is diagonal. In fact, since A A is positive semi-denite, all its eigenvalues are real and non-negative, and we can denote D = λ 2 λ 2 m, for some numbers 0 λ,..., R, called the Singular Values of A. We dene λ A A = W W. This notation is justied since positive semi-denite. ( A ) 2 A = A A. Note that A A is also self-adjoint and Why is the matrix A A is interesting? one reason is that for all v F m, Av 2 = Av, Av = v, A Av = v, A A A Av = A Av, A Av = A Av 2, that is, (.) Av = A Av. Besides the geometrical interpretation that A and A A have the same eect on a vector's length, this has the useful consequence, ker A = ker A A. As m = dim ker A + ranka = dim ker A A + rank A A, it also follows that We denote their common value by l. rank A A = ranka.

3 A PERSONAL INTERVIEW WITH THE SINGULAR VALUE DECOMPOSITION 3.2. Proof of the (Right) Polar Decomposition. Since A A is self-adjoint, there exists an orthonormal basis ψ,..., ψ m of F m whose elements are eigenvectors of A A. Using our previous notation for the eigenvalues of A A, suppose that A A ψ i = λ i ψ i for i =,..., m. For simplicity, let us assume that λ... > 0 (where m l = ranka) or in other words, that ψ l+,..., ψ m are basis vectors for ker A = ker A A (in case A has a non-trivial kernel). To factor A = U P, let P = A A and dene U M n m (F) as follows. Consider the vectors λ Aψ,..., Aψ l. This set is orthonormal: for i, j l, we have Aψ i, Aψ j = Aψ i, Aψ j = ψ i, A Aψ j = ψi, λ 2 λ j λ i λ j λ i λ j λ i λ j λ i λ jψ j = ψ i, ψ j = δ i,j j λ i (recall that λ,... are reals). Remark. In fact, we discovered that ψ l+,..., ψ m λ Aψ,..., Aψ l is an orthonormal basis for ImA. is an orthonormal basis for ker A, and We now use the assumption m n. that If l < m, take an orthonormal completion ϕ l+,..., ϕ m such that λ Aψ,..., Aψ l, ϕ l+,..., ϕ m constitutes an orthonormal set of vectors in F n. This is possible since m n. Finally, dene U = λ Aψ Aψ l ϕ l+ ϕ m ψ ψ2 M. n m (F). ψm Let us check that the matrix U has the desired properties. First, U has orthonormal columns: the left-hand matrix in the above product is unitary and hence has orthonormal columns, as does the right-hand matrix. It is simple to verify that generally, a product of matrices with orthonormal columns again has orthonormal columns. Next, Since 0 ψ. ψ2 0. ψ i = i, ψm 0. 0

4 A PERSONAL INTERVIEW WITH THE SINGULAR VALUE DECOMPOSITION 4 we nd that λ Uψ i = i Aψ i ϕ i i l l < i m so that λ i Uψ i i l UP ψ i = 0 Uψ i l < i m Aψ i i l = 0 ϕ i l < i m = Aψ i where we have used the fact that Aψ i = 0 for i = l +,..., m (these vector actually span ker A). Since the identity A = UP holds on the basis vectors, ψ,..., ψ m, we are done. Remark. If A is inverible, ker A = ker A A implies that A A is invertible. In this case, U is uniquely determined since U = A P. Remark. If A M n m (F) and m n, we can perform a right polar decomposition for A M m n (F). We obtain a matrix U M m n (F) with orthonormal columns such that A = U AA, or equivalently, A = AA U. Here AA is again positive semi-denite, but now U has orthonormal rows. This is the Left Polar Decomposition. If n = m, that is, if our matrix A is square, both decompositions are possible: there exist unitary matrices U, U M n n (F) such that A = U A A = AA U. It is interesting to note that t, in this case U = U holds if and only if A is normal (AA = A A). 2. The Singular Value Decomposition 2.. Roughly Speaking. At least two dierent decompositions go by the name of Singular Value Decomposition (SVD): () Any matrix A M n m (F) may be factorized as A = V D W, where V M n p (F) and W M m p (F) have orthonormal columns for p = min {m, n}, and D M p p (F) is diagonal with non-negative entries.

5 A PERSONAL INTERVIEW WITH THE SINGULAR VALUE DECOMPOSITION 5 (2) Any matrix A M n m (F) may be factorized as A = V D W, where V M n n (F) and W M m m (F) are unitary matrices, and D M n m (F) has non-negative entries on the main diagonal and zeros elsewhere. (The main diagonal of a matrix D M n m (F) is (D,..., D nn ).) We will work out SVD using the rst denition. Suppose that such a decompisition does exist, and denote V = v v p ; W = w w p ; D = d 0 0 d p. Since W W = V V = I p, for each i p, we have (2.) A A w i = W DV V DW w i = W D 2 W w i = W D 2 w. w p w i = d2 i w i and similarly (2.2) AA v i = V DW W DV v i = V D 2 V v i = V D 2 v. v p v i = d2 i v i. Should such a decomposition exist, then, the column w i of W must be an eigenvector of A A with eigenvalue d 2 i. Using notation from the previous section, if m n (so p = m) we must have simply W = ψ ψ m ; D = λ 0 0. Matlab, for example, uses the second one

6 A PERSONAL INTERVIEW WITH THE SINGULAR VALUE DECOMPOSITION SVD from the Polar Decomposition. Formally, the SVD is an easy consequence of the polar decomposition. Assume that m n (so p = m) and A M n m (F). We use the notations dened above, and employ the right polar decomposition: A = U A A = UW λ W = = = λ Aψ Aψ l ϕ l+ ϕ m λ Aψ Aψ l ϕ l+ ϕ m } {{ } V W W λ λ } {{ } D W = ψ. ; } ψm {{ } W Dene, then D = λ V = W = λ Aψ Aψ l ϕ l+ ϕ m ψ ψ m and we have, by denition, an SVD decomposition of A. Manifestly, (2.) holds. Let us convince ourselves that (2.2) also holds, namely, that the column v i of V is an eigenvectors of AA, which corresponds to the eigenvalue λ 2 i. Indeed, for i l we have AA v i = AA Aψ i = A (A Aψ i ) = A ( ( ) λ 2 ) i ψ i = λ 2 λ i λ i λ i Aψ i. i λ i

7 A PERSONAL INTERVIEW WITH THE SINGULAR VALUE DECOMPOSITION 7 For the case where l + i m, recall that by our construction ϕ l+,..., ϕ m (ImA). We always have (ImA) ker AA : indeed, x v (ImA), then for any u F n, 0 = AA u, v = u, AA v - which implies AA v = 0. Now, if l + i m then λ i = 0 and v i = ϕ i ker AA, so that AA v i = 0 = 0 ϕ i = λ 2 i ϕ i. Thus, in either case, the column v i of V is an eigenvectors of AA that corresponds to the eigenvalue λ 2 i.

linearly indepedent eigenvectors as the multiplicity of the root, but in general there may be no more than one. For further discussion, assume matrice

linearly indepedent eigenvectors as the multiplicity of the root, but in general there may be no more than one. For further discussion, assume matrice 3. Eigenvalues and Eigenvectors, Spectral Representation 3.. Eigenvalues and Eigenvectors A vector ' is eigenvector of a matrix K, if K' is parallel to ' and ' 6, i.e., K' k' k is the eigenvalue. If is