Constructing matrices with optimal block coherence

Transcription

1 Constructing matrices with optimal block coherence Andrew Thompson (Duke University) Joint work with Robert Calderbank (Duke) and Yao Xie (Georgia Tech) SIAM Annual Meeting, Chicago July

2 Outline Background: The subspace packing problem An equivalent notion: block coherence Who is interested in this?

3 Outline Background: The subspace packing problem An equivalent notion: block coherence Who is interested in this? Prior work: Lower bound on block coherence Deterministic constructions achieving the lower bound

4 Outline Background: The subspace packing problem An equivalent notion: block coherence Who is interested in this? Prior work: Lower bound on block coherence Deterministic constructions achieving the lower bound Our work: Almost optimal deterministic constructions with more subspaces Analysis of block coherence for random matrices

5 Subspace packing Let{S 1,S 2,...,S m } be r-dimensional subspaces of C n ; r n.

6 Subspace packing Let{S 1,S 2,...,S m } be r-dimensional subspaces of C n ; r n. S i G(n,r): Grassmann manifold

7 Subspace packing Let{S 1,S 2,...,S m } be r-dimensional subspaces of C n ; r n. S i G(n,r): Grassmann manifold Optimal packings: maximize the minimum distance between subspaces, for some distance metric: Grassmann packing

8 Subspace packing Let{S 1,S 2,...,S m } be r-dimensional subspaces of C n ; r n. S i G(n,r): Grassmann manifold Optimal packings: maximize the minimum distance between subspaces, for some distance metric: Grassmann packing LetA i C n r be an orthonormal basis for S i, i = 1,2,...,m.

9 Distance metrics Chordal distance: [d C (S i,s j )] 2 := r A ia j 2 F = r where {λ i } are the singular values of A ia j, λ 1 λ 2... λ r. r i=1 λ 2 i,

10 Distance metrics Chordal distance: [d C (S i,s j )] 2 := r A ia j 2 F = r where {λ i } are the singular values of A ia j, λ 1 λ 2... λ r. Spectral distance: r i=1 λ 2 i, [d S (S i,s j )] 2 := 1 A i A j 2 2 = 1 λ2 1.

11 Block coherence Let the matrix A be the concatenation of the orthonormal bases for {S 1,S 2,...,S m }: ] A = [A 1 A 2 A m, (1) A i C n r, i {1,2,...,m}, (2)

12 Block coherence Let the matrix A be the concatenation of the orthonormal bases for {S 1,S 2,...,S m }: ] A = [A 1 A 2 A m, (3) A i C n r, i {1,2,...,m}, (4) Define µ(a), the (worst-case) block coherence of A to be µ(a) := max i j A ia j 2.

13 Block coherence Let the matrix A be the concatenation of the orthonormal bases for {S 1,S 2,...,S m }: ] A = [A 1 A 2 A m, (5) A i C n r, i {1,2,...,m}, (6) Define µ(a), the (worst-case) block coherence of A to be µ(a) := max i j A ia j 2. Ifmr n,acan have orthonormal columns = µ(a) = 0.

14 Block coherence Let the matrix A be the concatenation of the orthonormal bases for {S 1,S 2,...,S m }: ] A = [A 1 A 2 A m, (7) A i C n r, i {1,2,...,m}, (8) Define µ(a), the (worst-case) block coherence of A to be µ(a) := max i j A ia j 2. Ifmr n,acan have orthonormal columns = µ(a) = 0. If2r > n, µ(a) = 1 since any two subspaces have nontrivial intersection.

15 Applications Needed in recovery guarantees for one-step group thresholding in compressed sensing: gives optimal order guarantees for deterministic matrices. Blind sensing of multiband signals DNA microarrays Medical imaging, e.g. ECG and EEG/MEG brain imaging Group model selection in statistics

16 Applications Needed in recovery guarantees for one-step group thresholding in compressed sensing: gives optimal order guarantees for deterministic matrices. Blind sensing of multiband signals DNA microarrays Medical imaging, e.g. ECG and EEG/MEG brain imaging Group model selection in statistics Special case: multiple measurement vector (MMV) model. Sensor networks and MIMO

17 Applications Needed in recovery guarantees for one-step group thresholding in compressed sensing: gives optimal order guarantees for deterministic matrices. Blind sensing of multiband signals DNA microarrays Medical imaging, e.g. ECG and EEG/MEG brain imaging Group model selection in statistics Special case: multiple measurement vector (MMV) model. Sensor networks and MIMO Grassmann packings: Multiple-antenna (MIMO) communication systems

18 Known bounds on packing distance Rankin bound for chordal distance: (Conway/Hardin/Sloane 1996) min i j [d C(S i,s j )] 2 r(n r) n m m 1, with equality if all pairs of subspaces are equidistant.

19 Known bounds on packing distance Rankin bound for chordal distance: (Conway/Hardin/Sloane 1996) min i j [d C(S i,s j )] 2 r(n r) n m m 1, with equality if all pairs of subspaces are equidistant. Bound for spectral distance: (Dhillon/Heath/Strohmer/Tropp 2008, Lemmens/Seidel 1973) min i j [d S(S i,s j )] 2 n r n m m 1, with equality if the subspaces are equi-isoclinic: all {λ i } among alla i A j for i j are equal in magnitude.

20 Lower bound on block coherence Lower bound on block coherence: (Lemmens/Seidel 1973) µ(a) mr n n(m 1), with equality if all {λ i } among alla ia j for i j are equal in modulus.

21 Lower bound on block coherence Lower bound on block coherence: (Lemmens/Seidel 1973) µ(a) mr n n(m 1), with equality if all {λ i } among alla ia j for i j are equal in modulus. Proof: The spectral distance bound gives 1 max i j A i A j 2 2 n r n which rearranges to give m m 1, max i j A ia j 2 2 n(m 1) m(n r) n(m 1) = mr n n(m 1).

22 Lower bound on block coherence Lower bound on block coherence: (Lemmens/Seidel 1973) µ(a) mr n n(m 1), with equality if all {λ i } among alla ia j for i j are equal in modulus. Proof: The spectral distance bound gives 1 max i j A i A j 2 2 n r n which rearranges to give m m 1, max i j A ia j 2 2 n(m 1) m(n r) n(m 1) Extends the Welch bound (1974) to r > 1. = mr n n(m 1).

23 Can the bound be achieved? For r = 1, the bound is achieved if A is an equiangular tight frame (ETF): (Welch 1974) The columns of A are unit norm. The inner products between pairs of different columns are equal in modulus. The columns form a tight frame, that is AA = (mr/n)i.

24 Can the bound be achieved? For r = 1, the bound is achieved if A is an equiangular tight frame (ETF): (Welch 1974) The columns of A are unit norm. The inner products between pairs of different columns are equal in modulus. The columns form a tight frame, that is AA = (mr/n)i. Several infinite families of ETFs are known (Tropp 2005, Fickus/Mixon/Tremain 2012, Jasper/Mixon/Fickus 2013).

25 Can the bound be achieved? For r = 1, the bound is achieved if A is an equiangular tight frame (ETF): (Welch 1974) The columns of A are unit norm. The inner products between pairs of different columns are equal in modulus. The columns form a tight frame, that is AA = (mr/n)i. Several infinite families of ETFs are known (Tropp 2005, Fickus/Mixon/Tremain 2012, Jasper/Mixon/Fickus 2013). For r > 1, some small examples are known (Conway/Hardin/Sloane 1996) and numerical methods have been proposed to approximately construct optimal packings (Dhillon/Heath/Strohmer/Tropp 2008).

26 An optimal construction Kronecker product construction: (Lemmens/Seidel 1973, CTX 2013) LetA = P Q wherep C (n/r) m is an ETF and Q C r r is a unitary matrix. Then the columns in each block are orthonormal, and µ(a) = mr n n(m 1).

27 An optimal construction Kronecker product construction: (Lemmens/Seidel 1973, CTX 2013) LetA = P Q wherep C (n/r) m is an ETF and Q C r r is a unitary matrix. Then the columns in each block are orthonormal, and mr n µ(a) = n(m 1). ] Writing P = [p 1 p 2... p m for the columns of P, the proof hinges on the fact that A ia j 2 = p i,p j.

28 An optimal construction Kronecker product construction: (Lemmens/Seidel 1973, CTX 2013) LetA = P Q wherep C (n/r) m is an ETF and Q C r r is a unitary matrix. Then the columns in each block are orthonormal, and mr n µ(a) = n(m 1). ] Writing P = [p 1 p 2... p m for the columns of P, the proof hinges on the fact that A ia j 2 = p i,p j. For every infinite family of ETFs (e.g. Steiner/Kirkman), this gives an infinite family of Grassmann packings.

29 How many subspaces in the packing? Subspace bound: (Lemmens/Seidel 1973) The number of r-dimensional equidistant subspaces in C n cannot exceedn 2 ; the number of equi-isoclinic subspaces in C n cannot exceedn 2 r 2 +1.

30 How many subspaces in the packing? Subspace bound: (Lemmens/Seidel 1973) The number of r-dimensional equidistant subspaces in C n cannot exceedn 2 ; the number of equi-isoclinic subspaces in C n cannot exceedn 2 r If the ETF P is(n/r) (n/r) 2, then we obtain m = (n/r) 2.

31 How many subspaces in the packing? Subspace bound: (Lemmens/Seidel 1973) The number of r-dimensional equidistant subspaces in C n cannot exceedn 2 ; the number of equi-isoclinic subspaces in C n cannot exceedn 2 r If the ETF P is(n/r) (n/r) 2, then we obtain m = (n/r) 2. Furthermore, the best known infinite families of ETFs scale asn n 3 2 = the number of subspaces is limited to (n/r) 3 2.

32 Almost optimal packings Unions of orthonormal bases: (CTX 2013) Suppose A is a union of unitary matrices. Then µ(a) r n.

33 Almost optimal packings Unions of orthonormal bases: (CTX 2013) Suppose A is a union of unitary matrices. Then µ(a) r n. Kronecker product construction: (CTX 2013) LetA = P Q wherep C (n/r) m is a concatenation of unitary matrices such that all inner products between columns in different blocks have magnitude r n, and where Q C r r is a unitary matrix. Then A is a concatenation of unitary matrices, and µ(a) = r n.

34 Almost optimal packings... Kronecker product construction 2: (CTX 2013) LetA = P Q wherep C (n/r) m is a concatenation of unitary matrices such that all inner products between columns in different blocks have magnitude r n, and where Q C r r is a unitary matrix. Then A is a concatenation of unitary matrices, and µ(a) = r n.

35 Almost optimal packings... Kronecker product construction 2: (CTX 2013) LetA = P Q wherep C (n/r) m is a concatenation of unitary matrices such that all inner products between columns in different blocks have magnitude r n, and where Q C r r is a unitary matrix. Then A is a concatenation of unitary matrices, and µ(a) = r n. Several families of such P exist: e.g. discrete chirp, Gabor, Kerdock.

36 Almost optimal packings... Kronecker product construction 2: (CTX 2013) LetA = P Q wherep C (n/r) m is a concatenation of unitary matrices such that all inner products between columns in different blocks have magnitude r n, and where Q C r r is a unitary matrix. Then A is a concatenation of unitary matrices, and µ(a) = r n. Several families of such P exist: e.g. discrete chirp, Gabor, Kerdock. P C (n/r) (n/r)2 = m = (n/r) 2.

37 Block coherence of random subspaces Suppose we havemrandom subspaces S i of R n, i = 1,2,...,m, each distributed i.i.d. uniformly on G(n,r).

38 Block coherence of random subspaces Suppose we havemrandom subspaces S i of R n, i = 1,2,...,m, each distributed i.i.d. uniformly ong(n,r). Each A i is a random orthogonal matrix: its distribution is rotation-invariant.

39 Block coherence of random subspaces Suppose we havemrandom subspaces S i of R n, i = 1,2,...,m, each distributed i.i.d. uniformly ong(n,r). Each A i is a random orthogonal matrix: its distribution is rotation-invariant. A i can be obtained as the singular vectors of a Gaussian matrix.

40 Block coherence of random subspaces Suppose we havemrandom subspaces S i of R n, i = 1,2,...,m, each distributed i.i.d. uniformly ong(n,r). Each A i is a random orthogonal matrix: its distribution is rotation-invariant. A i can be obtained as the singular vectors of a Gaussian matrix. We consider the distribution of(λ 1,λ 2,...,λ r ), the squared singular values of A ia j.

41 A joint distribution fora i A j (Absil/Edelman/Koev 2006) Ifn 2r, (λ 1,λ 2,...,λ r ) R r + follow the multivariate beta ( distribution Beta r r, ) n r 2 2, with pdf r f(λ 1,λ 2,...,λ r ) = c n,r (λ i λ j ) λ 1 2 i (1 λ i ) 1 2 (n 2r 1), where c n,r := i<j i=1 ( π 1 2 r2 Γ n ) r 2 [ ( r )] 2Γr ( n r ) Γr 2 2 and the multivariate gamma function Γ r (p) is defined as r Γ r (p) := π 1 4 r(r 1) Γ j=1 ( p+ 1 j ). 2

42 An asymptotic bound on µ(a) Let(m,n,r) such that r n β.

43 An asymptotic bound on µ(a) Let(m,n,r) such that r β. n Asymptotic bound: (CTX 2013) LetAbe formed from i.i.d. random orthogonal matrices. Given β (0,1/2), there exists a small constant â(β) such that P { [µ(a)] 2 â(β) β +ǫ } 0.

44 An asymptotic bound on µ(a) Let(m,n,r) such that r β. n Asymptotic bound: (CTX 2013) LetAbe formed from i.i.d. random orthogonal matrices. Given β (0,1/2), there exists a small constant â(β) such that P { [µ(a)] 2 â(β) β +ǫ } 0. â(β) is the solution in 2 a < 1/β to the equation ( ) 1 2β βlna+ ln(1 aβ) (1 β)ln(1 β) = 0. 2

45 An asymptotic bound on µ(a)... The factorâ(β) for random subspaces: theoretical upper bound (blue) and the empirical average from1000 trials with n = 1000 and m = (n/r) 2 (red).

46 Another analysis of chordal distance (Bodmann, 2013) Same random subspace model Shows that 1 r A ia j 2 F β There is no multiplicative constant This fits! Multivariate beta converges to a fixed distribution asymptotically: expected and largest squared singular values remain different.

47 Overview of proof 1 Bound the pdf of λ 1 by a univariate beta density...

48 Overview of proof 1 Bound the pdf of λ 1 by a univariate beta density... LetRbe the region of (r 1)-dimensional space λ i 0; i = 2,3,...,r, and given λ 1 > 0, letr λ1 be the sub-region of R consisting of all(λ 2,...,λ r ) such that λ 1 λ 2,λ 3,...,λ r 0. R λ1 f(λ 1 ) = c n,r λ (1 λ 1 ) 1 2 (n 2r 1) i<j (λ i λ j ) r i=2 λ 1 2 i (1 λ i ) 1 2 (n 2r 1) dλ i

49 Overview of proof 2 = f(λ 1 ) = c n,r λ r (1 λ 1 ) 1 2 (n 2r 1) R λ1 2 i<j (λ i λ j ) r i=2 λ 1 2 i (1 λ i ) 1 2 (n 2r 1) dλ i

50 Overview of proof 2 R = f(λ 1 ) c n,r λ r (1 λ 1 ) 1 2 (n 2r 1) 2 i<j (λ i λ j ) r i=2 λ 1 2 i (1 λ i ) 1 2 (n 2r 1) dλ i

51 Overview of proof 2 R = f(λ 1 ) c n,r λ r (1 λ 1 ) 1 2 (n 2r 1) 2 i<j (λ i λ j ) r i=2 λ 1 2 i (1 λ i ) 1 2 (n 2r 1) dλ i = f(λ 1 ) c n,r λ 1 2 (2r 1) 1 1 (1 λ 1 ) 1 2 (n 2r+1) 1. c n 2,r 1

52 Overview of proof 2 R = f(λ 1 ) c n,r λ r (1 λ 1 ) 1 2 (n 2r 1) 2 i<j (λ i λ j ) r i=2 λ 1 2 i (1 λ i ) 1 2 (n 2r 1) dλ i = f(λ 1 ) c n,r λ 1 2 (2r 1) 1 1 (1 λ 1 ) 1 2 (n 2r+1) 1. c n 2,r 1 This a multiple of a univariate beta pdf.

53 Overview of proof 3 = F(λ 1 ) π B[ 1 (2r 1), 1 (n 2r +1)] 2 2 [ ( B r, )] n r [ 1 I 1 λ1 2 (n 2r+1), 1 ] 2 (2r 1), where the regularized incomplete beta function (RIBF)I x (p,q) is I x (p,q) := 1 B(p, q) x 0 t p 1 (1 t) q 1 dt.

54 Overview of proof 3 = F(λ 1 ) π B[ 1 (2r 1), 1 (n 2r +1)] 2 2 [ ( B r, )] n r [ 1 I 1 λ1 2 (n 2r+1), 1 ] 2 (2r 1), where the regularized incomplete beta function (RIBF)I x (p,q) is I x (p,q) := 1 B(p, q) x 0 t p 1 (1 t) q 1 dt. Proportional-dimensional asymptotic for RIBF: (T 2012) Let0 < x < 1,p/(p+q) > x,p/(p+q) ρ. Then [ ( ρ ρln +(1 ρ)ln x) lim p 1 p+q lni x(p,q) = ( 1 ρ 1 x )].

55 Overview of proof 4 Write λ 1 = aβ...

56 Overview of proof 4 Write λ 1 = aβ... lim n ( ) 1 1 2β n ln F(aβ) βlna+ ln(1 aβ) (1 β)ln(1 β) 2 }{{} Ψ(a,β)

57 Overview of proof 4 Write λ 1 = aβ... lim n ( ) 1 1 2β n ln F(aβ) βlna+ ln(1 aβ) (1 β)ln(1 β) 2 }{{} Ψ(a,β) F(aβ) C e n Ψ(a,β).

58 Overview of proof 4 Write λ 1 = aβ... lim n ( ) 1 1 2β n ln F(aβ) βlna+ ln(1 aβ) (1 β)ln(1 β) 2 }{{} Ψ(a,β) F(aβ) C e n Ψ(a,β). Ifâ(β) solves Ψ(a,β) = 0, P(λ 1 > â(β) β +ǫ) is exponentially small.

59 Overview of proof 4 Write λ 1 = aβ... lim n ( ) 1 1 2β n ln F(aβ) βlna+ ln(1 aβ) (1 β)ln(1 β) 2 }{{} Ψ(a,β) F(aβ) C e n Ψ(a,β). Ifâ(β) solves Ψ(a,β) = 0, P(λ 1 > â(β) β +ǫ) is exponentially small. Union bounding over ( m 2) matricesa i A j, P([µ(A)] 2 > â(β) β +ǫ) 0.

60 Summary Optimal behaviour is essentially µ r n

61 Summary Optimal behaviour is essentially µ r n Our constructions give the best known number of blocks: (n/r) 2

62 Summary Optimal behaviour is essentially µ r n Our constructions give the best known number of blocks: (n/r) 2 Random subspaces also have µ r n asymptotically.

63 References Equi-isoclinic subspaces of Euclidean space; Lemmens, P. and Seidel, J. (Nederlandse Akademie van Wetenschappen, 1973) Packing lines, planes, etc.: packing in Grassmannian spaces; Conway, J., Hardin, R. and Sloane, N. (Experimental Mathematics, 1996) On the largest principal angle between random subspaces; Absil, P-S., Edelman, A. and Koev, P. (Linear Algebra and its Applications, 2006) Group model selection using marginal correlations: the good, the bad and the ugly; Bajwa, W.and Mixon, D. (Allerton Conference on Communication, Control and Computing, 2012) Random fusion frames are nearly equiangular and tight; Bodmann, B.(Linear Algebra and its Applications, 2013) On block coherence of frames; CTX (to appear, Applied & Computational Harmonic Analysis, 2014)

64 One-step group thresholding (OSGT) 1: Inputs: Measurement matrix ] A = [A 1 A 2 A m C n mr, 2: measurement vector y C n, group sparsity k. ] T [ ] T [f 1 f 2 f m A 1 A 2 A m y 3: ( I,{ f (j) 2 } ) SORT({ f i 2 })

65 Recovery result for OSGT Theorem: There exist positive constants c k and c ν such that, if c k rk < n, µ(a) c µ logm and rlogm ν(a) c ν µ(a), n the output of the one-step group thresholding algorithm satisfies FDP(ˆK) 1 L k ; NDP(ˆK) 1 L k, for an integer L which depends upon c µ and the l 2 -norms { x i 2 }.