Multi-stage convex relaxation approach for low-rank structured PSD matrix recovery

Transcription

1 Multi-stage convex relaxation approach for low-rank structured PSD matrix recovery Department of Mathematics & Risk Management Institute National University of Singapore (Based on a joint work with Shujun Bi and Shaohua Pan) May 22, 2014 / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 1 / 30

2 Outline 1 Problem formulation 2 Multi-stage convex relaxation approach Exact penalty for the equivalent MPSDEC A unified framework for multi-stage convex relaxation 3 The power of multi-stage convex relaxation approach General error bound Error bound comparison for the first two stages Error bound comparison for the first k stages 4 Numerical results 5 Conclusion / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 2 / 30

3 Outline 1 Problem formulation 2 Multi-stage convex relaxation approach Exact penalty for the equivalent MPSDEC A unified framework for multi-stage convex relaxation 3 The power of multi-stage convex relaxation approach General error bound Error bound comparison for the first two stages Error bound comparison for the first k stages 4 Numerical results 5 Conclusion / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 3 / 30

4 Matrix recovery Given some measurements for a (structured) matrix X H n + (Hermitian positive semi-definite), how to recover this unknown matrix? b k = Θ k, X + ξ k, k = 1, 2,..., m. b k : observed data (with / without noise) ξ k : additive noise Possible? Yes, with the low-rank structure. Entries = basis coefficients: {e i e T j 1 i n 1, 1 j n 2 }. / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 4 / 30

5 Correlation matrix recovery A low-rank correlation matrix X S n +: symmetric positive semidefinite and diag(x) = e. Observations (with / without noise): b k = Θ k, X + ξ k, k = 1,..., m. How to recover the unknown correlation matrix X? All diagonal entries are fixed. Some off-diagonal entries may also be fixed, [e.g., the correlations among pegged currencies]. Entries basis coefficients { e i e T i 1 i n } { 1 2 (e i e T j +e j e T i ) 1 i<j n }. / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 5 / 30

6 Density matrix recovery A low-rank density matrix X H n + with n = 2 l : Hermitian positive semidefinite and Tr(X) = 1. Observations Pauli measurements: Θ k, X = Re(Tr(Θ k X)), k = 1,..., m, where Θ k Pauli basis { σ s1 σ sl (s 1,, s l ) {0, 1, 2, 3} l} with σ 0 = ( ) 1 0, σ = How to recover the unknown density matrix X? ( ) ( ) ( ) , σ = 1 0, σ =. 0 1 / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 6 / 30

7 Observation model In this talk, we consider the problem of low-rank structured PSD matrix recovery with noise. The observation model takes the following form b k = Θ k, X + ξ k, k = 1,..., m. Define A(X) := ( Θ 1, X,, Θ m, X ) T R m. Then, we have that b = A(X) + ξ. Let X H n + be the unknown low-rank structured true matrix. There exists a constant δ 0, A(X) b δ. / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 7 / 30

8 The rank minimization estimator The rank minimization estimator: X arg min Ω H n + { rank(x) : A(X) b δ where Ω is a closed convex set characterizing the structure of X. This is the best possible estimator. But it is very challenging to get the global solution due to the nonconvexity and discontinuity of the rank function. } / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 8 / 30

9 The nuclear norm technique The nuclear norm technique has gained success in many applications: n rank(x) = X := σ i (X), where σ 1 (X) σ n (X) denote all the singular values of X. Nuclear norm convex envelope of the rank function over the unit ball of the spectral norm. y i=1 x /a 1 rank x a 0 a x / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 9 / 30

10 Nuclear norm estimator Nuclear norm estimator: X arg min Ω H n + { } X : A(X) b δ Extra biased (for large singular values). It may fail when certain rows and / or columns are heavily sampled (Salakhutdinov and Srebro, 2010). It may not be rank consistent for general sampling even for the low-dimensional case (Bach, 2008). For correlation and density matrices, X = constant = the nuclear norm technique s power is limited. / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 10 / 30

11 Outline 1 Problem formulation 2 Multi-stage convex relaxation approach Exact penalty for the equivalent MPSDEC A unified framework for multi-stage convex relaxation 3 The power of multi-stage convex relaxation approach General error bound Error bound comparison for the first two stages Error bound comparison for the first k stages 4 Numerical results 5 Conclusion / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 11 / 30

12 Equivalent MPSDEC Let C(J) be the family of closed convex functions φ : J R with J [0, 1] and φ(0) = 0, 0 < t := arg min 0 t 1 φ(t) and φ (0) < +. Rank minimization estimator is equivalent to the following mathematical program with the PSD equilibrium constraint (MPSDEC) min I, Φ(W) X Ω,W Hn s.t. A(X) b δ (1) W, X = 0, X 0, 0 W I Φ the Löwner operator associated to φ C(J) The bilinear constraint W, X = 0 is the trouble maker. / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 12 / 30

13 Exact penalty for MPSDEC To deal with W, X = 0 efficiently, we consider the penalized version min X Ω,W Hn I, Φ(W) + ρ W, X s.t. A(X) b δ (2) X 0, 0 W I Theorem The cost function of (2) has a desired structure though it is nonconvex The penalty problem (2) is exact in the following sense There exists ρ > 0 such that when ρ > ρ, the set of global optimal solutions of the penalty problem (2) coincides with that of the MPSDEC. / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 13 / 30

14 Multi-stage convex relaxation approach (S0) Select a function φ C(J). Let W 0 := I and set k := 1. Choose ρ 0 > 0. (S1) Solve the following weighted trace-norm minimization problem X k arg min X Ω H+ n { W k 1, X : A(X) b δ }. (S2) By the information of X k select a suitable ρ k = δ k ρ k 1 with δ k 1. (S3) Solve the following minimization problem W k arg min 0 W I (S4) Set k := k + 1, and go to (S.1). { I, Φ(W) + ρ k W, X k }. / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 14 / 30

15 Multi-stage convex relaxation (Examples) The convex relaxation approach is solving (2) in an alternating way. The computational cost is the solution of the weighted trace-norm minimization problem since if X k = U k Diag(λ(X k ))(U k ) T we have W k = U k Diag(w k 1,..., wk n)(u k ) T. 1 φ 1 (t) = t w k i = { 0 if λi (X k ) 1 ρ k 1 otherwise 2 φ 2 (t) = a2 1 4 t 2 at w k i = min( 2 a 2 1 max(a ρ kλ i (X k ), 0), 1). 3 φ 3 (t) = t + 1 t+ɛ 1 ɛ w k 1 i = max( ɛ, 0). ρk λ i (X k )+1 4 φ 4 (t) = t ln(t + ɛ) + ln(ɛ) w k 1 i = max( ρ k λ i ɛ, 0). (X k )+1 / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 15 / 30

16 Outline 1 Problem formulation 2 Multi-stage convex relaxation approach Exact penalty for the equivalent MPSDEC A unified framework for multi-stage convex relaxation 3 The power of multi-stage convex relaxation approach General error bound Error bound comparison for the first two stages Error bound comparison for the first k stages 4 Numerical results 5 Conclusion / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 16 / 30

17 Notation and assumptions Recall that rank(x) = r. Assume that X = [U 1, U 2 ]Diag(λ(X))[U 1, U 2 ] T, where U 1 O n r and U 2 O n (n r). For any X H n, define P T (X) = U 1 U T 1 X + XU 1 U T 1 U 1 U T 1 XU 1 U T 1. Let Q = A A. Define the restricted eigenvalues of Q by (Tong Zhang, 2010) X, Q(X) X, Q(X) ϑ + (k) := sup 0<rank(X) k X 2 F, ϑ (k) := inf 0<rank(X) k X 2 F Assumption: There exist a c [0, 1) and an integer s (1 s n 2r 2 ) s.t. ϑ + (s) ϑ (2r + 2s) 1 + 2c2 s. r / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 17 / 30

18 General error bound Theorem Let γ k 1 := P T(W k 1 ) λ min ((U 2 ) T W k 1 U 2 ). If 0 γ k 1 1 c, then Xk X F Ξ(γ k 1 ) with Ξ(t) := 1 + rt2 2s 2 ϑ + (2r + s)δ (1 ct) ϑ (2r + s) Ξ(1) the error bound for the nuclear norm estimator. 5 2 ) 3 2 (1 δ 3r) If c < 1 1 δ 3r(1+ Ξ(1) = for t 0., then we have the following inequality 6 ϑ+ (3r)δ (1 c) ϑ (3r) δ3r δ (1 c) (1 δ 3r ) < 3δ 1 + δ3r 1 δ 3r (1 + The last term is the bound given by Mohan and Fazel (2010). 5 2 ). / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 18 / 30

19 Error bounds in the first two stages Let ψ be the conjugate function of ψ(t) := { φ(t) if t [0, 1], + otherwise. Define a 0 := (ψ ) +(0), a 1 := (ψ ) ( ρ1 λ r+1 (X 1 ) ) and b 1 := (ψ ) +( ρ1 λ r (X 1 ) ). Theorem Suppose that λ r (X) = β X 1 X F for some β > 2. If the parameter ρ 1 is chosen such that ρ 1 ( 0, φ + (0) ) b 1 + a 0 β ln λ r+1 (X 1 ) and 2 β 2 < 1, then a 1 a 1 2 ln β β 2 X 2 X F Ξ(γ 1 ) Ξ b 1 + a 0 a 1 a 1 β 2 ln β 2 β 2 ln β 2 < Ξ(1). (3) / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 19 / 30

20 Reduction ratio given by different φ Table: The ratio of the error bounds given by different φ for the first two stages φ 1 ( φ 2 ( φ 3 ( ( φ 4 ( φ 4 ρ 1 β ratio ) Ξ(γ ) λ r+1 (X 1 ) Ξ(1) ) Ξ(γ ) λ r+1 (X 1 ) Ξ(1) ) Ξ(γ ) λ r+1 (X 1 ) Ξ(1) ) Ξ(γ ) λ r+1 (X 1 ) Ξ(1) ) Ξ(γ ) Ξ(1) λ r(x 1 ), 1 2 λ r(x 1 ), 1 3 λ r(x 1 ), λ r(x 1 ), λ r(x 1 ), 0.8 λ r+1 (X 1 ) / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 20 / 30

21 Guaranteed error bound reduction (Cont.) Let b k := (ψ ) +[ ρk (λ r (X ) Ξ( γ k 1 )) ], ã k := (ψ ) ( ρ k Ξ( γ k 1 )), γ k := b k + a 0 2 ln ã k ãk 2 ln λ r(x) λ r(x) 2Ξ( γ k 1 ) λ r(x) λ r(x) 2Ξ( γ k 1 ) for k 1 and γ 0 = 1. Theorem Suppose that λ r (X) = βξ(1) for some β > 2. If the parameter ρ 1 is chosen such that ρ 1 ( 0, φ + (0) ) b 1 + a 0 β ln Ξ(1) and 2 β 2 < 1, and the parameters σ k are chosen ã 1 ã1 2 ln β β 2 such that σ k [ 1, Ξ( γ k 2) Ξ( γ k 1 )], then for all k 2 we have γk 1 γ k 1 < γ k 2 and X k X F Ξ(γ k 1 ) Ξ( γ k 1 ) < Ξ( γ k 2 ). (4) / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 21 / 30

22 Reduction ratio of the error bounds Table: The ratio of the error bounds obtained with φ 2 for the first five stages ( φ Ξ(1), 1.2 ) Ξ(1) ρ 1 ratio Ξ(γ 1 )/Ξ(1) Ξ(γ 2 )/Ξ(1) Ξ(γ 3 )/Ξ(1) Ξ(γ 4 )/Ξ(1) Ξ(γ 5 )/Ξ(1) / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 22 / 30

23 Outline 1 Problem formulation 2 Multi-stage convex relaxation approach Exact penalty for the equivalent MPSDEC A unified framework for multi-stage convex relaxation 3 The power of multi-stage convex relaxation approach General error bound Error bound comparison for the first two stages Error bound comparison for the first k stages 4 Numerical results 5 Conclusion / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 23 / 30

24 The performance with a high sample ratio φ 1 φ 2 with a=2.1 φ 3 with ε=0.4 φ 4 with ε= φ 1 φ 2 with a=2.1 φ 3 with ε=0.4 φ 4 with ε= Relative error (%) Computing time (in seconds) (a) The number of stages (b) The number of stages Figure: The performance of multi-stage convex relaxation with p = / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 24 / 30

25 The performance with a low sample ratio φ 1 φ 2 with a=2.1 φ with ε=0.4 3 φ with ε= φ 1 φ 2 with a=2.1 φ with ε=0.4 3 φ with ε= Relative error (%) Computing time (in seconds) (a) The number of stages (b) The number of stages Figure: The performance of multi-stage convex relaxation with p = 3980 / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 25 / 30

26 Low-rank correlation matrix recovery Table: Performance for the correlation matrix recovery problems with n = 1000 r 5 10 off diag sr(%) One-stage Two-stage Final result relerr(rank) time relerr(rank) time k relerr(rank) time e-1(1000) e-1(19) e-1(5) e-1(998) e-1(5) e-1(5) e-2(999) e-2(5) e-2(5) e-1(991) e-1(18) e-1(5) e-1(977) e-1(5) e-1(5) e-1(703) e-2(6) e-2(5) e-1(1000) e-1(21) e-1(10) e-1(1000) e-1(10) e-1(10) e-2(1000) e-2(10) e-2(10) e-1(1000) e-1(17) e-1(10) e-1(1000) e-2(10) e-2(10) e-2(991) e-2(10) e-2(10) 787 / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 26 / 30

27 Low-rank density matrix recovery Table: Performance for the low-rank density matrix recovery problems r n sr(%) One-stage Two-stage Final result relerr(rank) time relerr(rank) time k relerr(rank) time e-1(31) e-1(3) e-1(3) e-1(31) e-1(3) e-1(3) e-1(31) e-1(3) e-2(3) e-1(41) e-2(3) e-2(3) e-1(41) e-1(3) e-2(3) e-1(40) e-2(3) e-2(3) e-1(42) e-1(5) e-1(5) e-1(43) e-1(5) e-1(5) e-1(43) e-1(17) e-1(5) 1014 / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 27 / 30

28 Low-rank covariance matrix recovery Table: Performance for the low-rank covariance matrix recovery with n = 1000 r nfixed sr(%) One-stage Two-stage Final result relerr(rank) time relerr(rank) time k relerr(rank) time (200, 0) e-1(58) e-1(10) e-1(10) 578 (0, 200) e-1(59) e-1(10) e-1(10) 630 (200, 200) e-1(57) e-1(10) e-1(10) 580 (200, 0) e-1(52) e-2(10) e-2(10) 444 (0, 200) e-1(49) e-1(10) e-2(10) 329 (200, 200) e-1(53) e-2(10) e-2(10) 463 (200, 0) e-1(86) e-1(20) e-1(20) 346 (0, 200) e-1(88) e-1(20) e-1(20) 322 (200, 200) e-1(87) e-1(20) e-1(20) 415 (200, 0) e-1(79) e-1(20) e-1(20) 270 (0, 200) e-1(78) e-1(20) e-1(20) 261 (200, 200) e-1(80) e-1(20) e-1(20) 294 / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 28 / 30

29 Outline 1 Problem formulation 2 Multi-stage convex relaxation approach Exact penalty for the equivalent MPSDEC A unified framework for multi-stage convex relaxation 3 The power of multi-stage convex relaxation approach General error bound Error bound comparison for the first two stages Error bound comparison for the first k stages 4 Numerical results 5 Conclusion / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 29 / 30

30 Conclusion We proposed a multi-stage convex relaxation approach for the rank minimization problem by solving the exact penalty problem of its equivalent MPSDEC in an alternating way. Under a weaker condition than the RIP, we established a tighter error bound for the optimal solution of each subproblem to the global optimal solution of the original problem, and provided a quantitative analysis for the decrease of the error bound. The two-stage convex relaxation can improve the error bound given by the trace norm relaxation method at least 30% for those not nice problems. The general case can be done by using the theory on spectral operators of matrices. / Defeng Sun /NUS Adaptive nuclear semi-norm penalization OP14, San Diego 30 / 30