ENGG5781 Matrix Analysis and Computations Lecture 9: Kronecker Product Wing-Kin (Ken) Ma 2017 2018 Term 2 Department of Electronic Engineering The Chinese University of Hong Kong
Kronecker product and properties vectorization Kronecker sum Lecture 9: Kronecker Product W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 1
Motivating Problem: Matrix Equations Problem: given A, B, find an X such that AX = B. ( ) an easy problem; if A has full column rank and ( ) has a solution, the solution is merely X = A B. Question: but how about matrix equations like AX + XB = C, A 1 XB 1 + A 2 XB 2 = C, AX + YB = C, X, Y both being unknown? such matrix equations can be tackled via matrix tools arising from the Kronecker product W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 2
Kronecker Product The Kronecker product of A and B is defined as Example: A B = a 11 B a 12 B... a 1n B a 21 B. a 22 B... a 2n B. a m1 B a m2 B... a mn B let a R m, b R m. By definition, a 1 b a b = a 2 b. a m b Note that, since ba T = [ a 1 b, a 2 b,..., a m b ], a b is a column-by-column concatenation of the outer product ba T.. W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 3
Properties Elementary properties: 1. A (αb) = (αa) B. 2. (A + B) C = A C + B C, A (B + C) = A B + A C (distributive) 3. A (B C) = (A B) C (associativity). 4. 0 mn = 0 m 0 n, I mn = I m I n ; 0 n and I n are n n zero and identity matrices. 5. (A B) T = A T B T, (A B) H = A H B H. 6. there exist permutation matrices U 1 and U 2 such that U 1 (A B)U 2 = B A. Note: Kronecker product is not commutative; i.e., A B B A in general. Property 6 above is a weak version of commutativity. W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 4
Property 9.1 (mixed product rule). More Properties (A B)(C D) = (AC) (BD), for A, B, C, D of appropriate matrix dimensions. Some properties from Property 9.1: 1. if A R m m and B R n n are nonsingular, then (A B) 1 = A 1 B 1 proof: (A 1 B 1 )(A B) = (A 1 A) (B 1 B) = I m I n = I mn. 2. if Q 1, Q 2 are semi-orthogonal, then Q 1 Q 2 is semi-orthogonal. proof: (Q 1 Q 2 ) T (Q 1 Q 2 ) = (Q T 1 Q T 2 )(Q 1 Q 2 ) = (Q T 1 Q 1 ) (Q T 1 Q 1 ) = I. W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 5
Example: Hadamard Matrix Consider an 2 2 orthogonal matrix H 2 = 1 [ ] 1 1 2 1 1. From H 2, construct a 4 4 matrix H 4 = H 2 H 2 = 1 2 1 1 1 1 1 1 1 1 1 1 1 1, 1 1 1 1 and inductively, H n = H n/2 H n/2 for any n that is a power of 2. is H 4 orthogonal? Yes, because H 4 H T 4 = (H 2 H 2 )(H T 2 H T 2 ) = (H 2 H T 2 H 2 H T 2 ) = I. for the same reason, any H n is orthogonal W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 6
Kronecker Product and Eigenvalues Theorem 9.1. Let A R m m, B R n n. Let {λ i, x i } m i=1 be the set of m eigen-pairs of A, and let {µ i, y i } n i=1 be the set of n eigen-pairs of B. The set of mn eigen-pairs of A B is given by {λ i µ j, x i y j } i=1,...,m, j=1,...,n Properties arising from Theorem 9.1 (for square A, B): 1. det(a B) = [det(a)] n [det(b)] m. 2. tr(a B) = tr(a)tr(b). 3. if A and B are (symmetric) PSD, then A B is PSD. W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 7
Vectorization The vectorization of A R m n is defined as a 1 a n vec(a) = a 2., i.e., we stack the columns of a matrix to form a column vector. Property 9.2. vec(axb) = (B T A)vec(X). Special cases of Property 9.2: vec(ax) = (I A)vec(X) vec(xa) = (A T I)vec(X) W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 8
write Proof Sketch of Property 9.2 m n X = x ij e i e T j i=1 j=1 by letting a i be the ith column of A and b j the jth row of B, m n m n vec(axb) = vec x ij Ae i e T j B = x ij vec(a i b T j ). i=1 j=1 i=1 j=1 by noting vec(a i b T j ) = vec([ a i b j1,..., a i b j,q ]) = b j1a i. = b j a i b j,q a i we get vec(axb) = m i=1 n j=1 x ijb j a i = (B T A)vec(X). W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 9
Kronecker Sum Problem: given A R n n, B R m m, C R n m, solve AX + XB = C ( ) with respect to X R m n. the above problem is a linear system. By vectorizing ( ), we get (I m A)vec(X) + (B T I n )vec(x) = vec(c) the Kronecker sum of A R n n and B R m m is A B = (I m A) + (B I n ). if a unique solution to ( ) is desired, we wish to know conditions under which A B is nonsingular W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 10
Kronecker Sum Theorem 9.2. Let {λ i, x i } n i=1 be the set of n eigen-pairs of A, and let {µ i, y i } m i=1 be the set of m eigen-pairs of B. The set of mn eigen-pairs of A B is given by {λ i + µ j, y j x i } i=1,...,n, j=1,...,m Theorem 9.3. The matrix equations AX + XB = C has a unique solution for every given C if and only if λ i µ j, for all i, j, where {λ i } n i=1 and {µ i} m i=1 are the set of eigenvalues of A and B, resp. idea behind Theorem 9.3: if λ i = µ j for some i, j, then from Theorem 9.2 there exists a zero eigenvalue for A B. W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 11
Kronecker Sum Consider A T X + XA = C, which is called the Lyapunov equations. from Theorem 9.3, the Lyapunov equations admit a unique solution if λ i λ j, for all i, j. if A is PD such that λ i > 0 for all i, the Lyapunov equations always have a unique solution. W.-K. Ma, ENGG5781 Matrix Analysis and Computations, CUHK, 2017 2018 Term 2. 12