Semidefinite Relaxations Approach to Polynomial Optimization and One Application. Li Wang

Transcription

1 Semidefinite Relaxations Approach to Polynomial Optimization and One Application Li Wang University of California, San Diego Institute for Computational and Experimental Research in Mathematics September 9, 2014 Li Wang (UCSD) SDP relaxations to POP and Application September 9, / 27

2 Outline Introduction: Semidefinite Relaxations for Polynomial Optimization Application: SDP Relaxations for Tensor Best Rank-1 Approximations Li Wang (UCSD) SDP relaxations to POP and Application September 9, / 27

3 Outline Introduction: Semidefinite Relaxations for Polynomial Optimization Application: SDP Relaxations for Tensor Best Rank-1 Approximations Li Wang (UCSD) SDP relaxations to POP and Application September 9, / 27

4 Polynomial Optimization Problem Consider the following problem: 8 f >< min := min f(x) x2r n s.t. h 1 (x) = = h m1 (x) =0, >: g 1 (x) 0,...,g m2 (x) 0. This problem is NP-hard even if f(x) is nonconvex quadratic and all constraints are linear. Standard Method: Lasserre s SDP Relaxation Li Wang (UCSD) SDP relaxations to POP and Application September 9, / 27

5 Sum of Squares (SOS) A polynomial f is SOS if it is a sum of squares of other polynomials, i.e., f = rx qj 2. Let f be a degree 2d polynomial, f is SOS if and only if there exists X 0, suchthat j=1 f(x) =[x] T d X[x] d = X ([x] d [x] T d ). Here [x] d denotes the vector of monomials [x] d =[1 x 1 x n x 2 1 x 1 x 2 x d n] T. The length of [x] d is N = n+d d. Li Wang (UCSD) SDP relaxations to POP and Application September 9, / 27

6 An SDP formulation of Sum of Squares Define 0/1 constant symmetric matrices C and A in the way that [x] d [x] T d = C + X deg(x )apple2d A x. If f(x) = P f x,thenf(x) is SOS if and only if C X = f 0, A X = f, 8 2 N n :0< apple2d X 0 M d (y) is called moment matrix of order d defined as: M d (y) := C + X deg(x )apple2d A y. Li Wang (UCSD) SDP relaxations to POP and Application September 9, / 27

7 Lasserre s SDP Relaxation: SOS version Let h := (h 1,,h m1 ), g := (g 0,g 1,...,g m2 ) where g 0 =1. The k-th truncated quadratic module generated by (h, g) is defined as 8 9 < Xm 1 Xm 2 i are SOS, j 2 R[x], 8 i, j = Q k (h, g) := jh j + ig i : deg( i g i ) apple 2k, deg( j h j ) apple 2k;. j=1 i=0 The k-th Lasserre s SOS relaxation (k is also called the relaxation order) is f k := max s.t. f(x) 2 Q k (h, g). It is equivalent to an SDP problem. Li Wang (UCSD) SDP relaxations to POP and Application September 9, / 27

8 Lasserre s SDP Relaxation: Moment Version The dual problem of SOS relaxation is: 8 fk := min hf,yi >< y2m 2k s.t. L (k d h j ) h j (y) =0, j 2 [m 1 ], L (k dg i ) g i (y) 0, i 2 [m 2 ], >: M k (y) 0, h1,yi =1. Here L (k d h j ) h j (y) and L (k dg i ) g i (y) are the localizing moment matrix, which is linear in y, i.e. L (k d h j ) h j (y) := X A y. Li Wang (UCSD) SDP relaxations to POP and Application September 9, / 27

9 Convergence of Lasserre s Hierarchy Theorem (Lasserre, 2001) If the archimedean condition holds, as k!1, lim f k = lim f k = f min. k!1 k!1 Lasserre s hierarchy has finite convergence if f k = f min for some order k<1. Under the Archimedean Condition, Lasserre s SDP relaxation has finite convergence generically. Li Wang (UCSD) SDP relaxations to POP and Application September 9, / 27

10 Finite Convergence Certificate Let y be an optimizer of moment version of Lasserre s SDP relaxation of order k. If the flat truncation condition (FTC) holds, i.e., rank M t d (y )=rank M t (y ). for some integer t 2 [ ˆd, k], and ˆd = max{d f,d}, thenf k = f min. The FTC is generically satisfiable if the polynomial optimization problem has finitely many minimizers. Li Wang (UCSD) SDP relaxations to POP and Application September 9, / 27

11 Outline Introduction: Semidefinite Relaxations for Polynomial Optimization Application: SDP Relaxations for Tensor Best Rank-1 Approximations Li Wang (UCSD) SDP relaxations to POP and Application September 9, / 27

12 What Is Tensor? A tensor of order m and dimension (n 1,...,n m ) is an array F that is indexed by integer tuples (i 1,...,i m ) with 1 apple i j apple n j for j =1,...,m, i.e., F =(F i1,...,i m ) 1applei1 applen 1,...,1applei mapplen m. Tensors of order m are called m-tensors. For example, a given tensor F2R : F = Here F 123 =4, F 121 =2, F 322 =9, F 312 =3. Li Wang (UCSD) SDP relaxations to POP and Application September 9, / 27

13 Symmetric Tensors AtensorF2R n 1 n m is symmetric if n 1 = = n m and F i1,...,i m = F j1,...,j m for all (i 1,...,i m ) (j 1,...,j m ), where means that (i 1,...,i m ) is a permutation of (j 1,...,j m ). For example, a given symmetric tensor F2R : F = F 111 =1, F 121 = F 211 = F 112 =2,, F 333 = 10. Li Wang (UCSD) SDP relaxations to POP and Application September 9, / 27

14 Outer Product and Tensor Decomposition Given vectors u 1,...,u m,defineu 1 u m as (u 1 u m ) i1,...,i m =(u 1 ) i1 (u m ) im. It is a rank-1 tensor in R n 1 n m. For every F of order m, thereexistu i,j 2 C n j,suchthat F = rx u i,1 u i,m. i=1 The smallest r is the rank of F, and is denoted as rank (F). It is hard to get a rank decomposition of F. Li Wang (UCSD) SDP relaxations to POP and Application September 9, / 27

15 Best Rank-1 Approximation for Tensors Given a tensor F2R n 1 n m,atensorb is a best rank-1 approximation of F if it is a minimizer of 8 < min kf X k 2 X2R n 1 nm : s.t. rank X =1. We can express B as B = u 1 u m, for 2 R and u 1 2 R n 1,,u m 2 R nm. Li Wang (UCSD) SDP relaxations to POP and Application September 9, / 27

16 Tensors and Polynomials Given a tensor F2R n 1 n m, define the polynomial F (x 1,...,x m ):= X F i1,...,i m (x 1 ) i1 (x m ) im, i 1,...,i m Note: F (x 1,...,x m ) is homogenous in each vector x i 2 R n i ; F (x 1,...,x m ) is multi-linear. Li Wang (UCSD) SDP relaxations to POP and Application September 9, / 27

17 Characterization of Rank-1 Approximation Theorem: (De Lathauwer, De Moor and Vandewalle, 2000) For a tensor F, the rank-1 approximation problem is equivalent to ( max F (x 1,...,x m ) s.t. kx 1 k 2 = = kx m k 2 =1, that is, B is a best rank-1 approximation for F if and only if B = (u 1 u m ), where (u 1,...,u m ) is a global maximizer of the above problem and = F (u 1,...,u m ). Li Wang (UCSD) SDP relaxations to POP and Application September 9, / 27

18 Rank-1 Approximation For Symmetric Tensors Given a symmetric F, B is a best rank-1 approximation of F if where u is the global maximizer of: B = (u u) and = f(u), f(x) = max f(x) s.t. kxk 2 =1, x2r n X 1applei 1,...,i mapplen F i1,,i m (x) i1 (x) im. So, we need to solve two polynomial optimization problems: (I) max f(x), (II) max f(x). x2sn 1 x2s n 1 Li Wang (UCSD) SDP relaxations to POP and Application September 9, / 27

19 Semidefinite Relaxation (I): Even Order m =2d Consider the following optimization problem: 8 max f(x) = X f x 1 1 x n n >< >: s.t. 2N n m (x T x) d = X 2N n m [x d ][x d ] T = X where [x d ] is the monomial vector: 2N n m g x 1 1 x n n =1, A x 1 1 x n n 0, [x d ]= x d 1 x d 1 1 x 2 x d 1 1 x n x d n T, N n m = {( 1,, n ) 2 N n n = m}. Li Wang (UCSD) SDP relaxations to POP and Application September 9, / 27

20 Semidefinite Relaxation (II): Even Order m =2d Replace each monomial x by a variable y and get : 8 X f y >< max y2r Nn m 2N n m s.t. hg, yi := X g y =1, >: 2N n m M(y) := X 2N n m A y 0. This is an SDP problem. It can be solved numerically, e.g., Interior point methods (small to moderate sizes); Regularization methods (large sizes). Li Wang (UCSD) SDP relaxations to POP and Application September 9, / 27

21 Semidefinite Relaxation (III): Even Order m =2d Let y be the maximizer of SDP problem: 8 X max f y y2r >< Nn m 2N n m X s.t. g y =1, >: 2N n m M(y) 0. If rank M(y )=1,thereexistsu 2 S n 1 such that y =[(u ) m ] and ku k 2 =1, and u is a global maximizer of max f(x) s.t. kxk 2 =1. So B = f(u ) (u u ) is a best rank-1 approximation. Li Wang (UCSD) SDP relaxations to POP and Application September 9, / 27

22 Semidefinite Relaxation (IV): Even Order m =2d Let y be the solution of SDP relaxation. If rank M(y ) > 1, wearenot guaranteed to get best rank-1 approximation. In this case, choose û as: (for some proper chosen s 2 [1,n]) ũ =(y (2d 1)e s+e 1,...,y (2d 1)e s+e n ), û =ũ/kũk. Then we use local optimization method to solve by using û as a starting point. max f(x) s.t. kxk 2 =1, x2r n Similarly, we repeat the above process to solve the problem max f(x) s.t. x T x =1. Li Wang (UCSD) SDP relaxations to POP and Application September 9, / 27

23 Algorithm: Rank-1 approximations for even symmetric tensors Step 1: Solve the semidefinite relaxations of: (I) max f(x) and (II) max f(x). x2sn 1 x2s n 1 Step 2: Let y and z be the solutions of SDP relaxations for (I)-(II), and choose ũ, ṽ from the SDP solutions; Step 3: If rank M(y )=1and rank M(z )=1, we get best rank-1 approximation; Step 4: If max{rank M(y ), rank M(z )} > 1, thenapplyalocal optimization method to improve the solutions. Li Wang (UCSD) SDP relaxations to POP and Application September 9, / 27

24 Measure of Approximation Quality Measurement: SDP relaxation is tight or not, we get an upper bound: The error f ubd := max{ f sdp max, f sdp min }. aprxerr := f(u) f ubd / max{1,f ubd } is a measure of the approximation quality. Local Optimization Method: Use Matlab optimization ToolBox: if SDP relaxation is not tight. fmincon to improve the solution Li Wang (UCSD) SDP relaxations to POP and Application September 9, / 27

25 Example 1 (Kolda and Mayo, 2011) (Zhang et al., 2012) Consider symmetric tensor F2R : F 111 = , F 112 =0.0516, F 113 = , F 122 = , F 123 = , F 133 = , F 222 =0.3251, F 223 = , F 233 = , F 333 = By semidefinite relaxation method, we get the rank-1 tensor B = u 3 with =0.8730, u =( , , ). The M(y ) has rank one, so u 3 is a best rank-1 approximation. The error aprxerr=1.2e-7. Li Wang (UCSD) SDP relaxations to POP and Application September 9, / 27

26 Example 2: Symmetric Random Example We generate symmetric tensor F2R n n of order m, witheachentry being a random variable obeying Gaussian distribution (randn in Matlab). (n, m) (N,M) time (min,med,max) aprxerr (min,med,max) (10,3) (66,1000) 0:00:01 0:00:01 0:00:03 (7.9e-9, 4.5e-8, 2.9e-6) (20,3) (231,10625) 0:00:03 0:00:08 0:00:13 (2.4e-9, 3.6e-7, 4.3e-6) (30,3) (496,46375) 0:01:14 0:01:29 0:02:01 (9.1e-9, 7.4e-7, 1.4e-5) (40,3) (861,135750) 0:06:32 0:10:04 0:13:09 (1.3e-9, 4.6e-6, 2.3e-3) (50,3) (1326,316250) 0:12:39 0:13:34 0:14:01 (3.2e-9, 1.3e-6, 2.0e-3) (15,4) (120,3060) 0:00:01 0:00:03 0:00:04 (4.0e-9, 1.1e-7, 1.3e-6) (20,4) (210,8854) 0:00:52 0:01:09 0:01:25 (1.2e-8, 1.8e-7, 6.3e-3) (25,4) (325,20475) 0:00:30 0:00:35 0:00:56 (4.7e-9, 1.3e-7, 1.0e-5) (30,4) (465,40919) 0:06:03 0:07:36 0:09:31 (1.2e-8, 1.1e-6, 9.6e-4) (35,4) (630,73815) 0:02:46 0:04:57 0:06:54 (4.1e-8, 1.6e-7, 7.4e-3) (10,5) (286,8007) 0:00:08 0:00:14 0:00:17 (4.3e-8, 4.1e-7, 4.1e-6) (15,5) (816,54263) 0:03:46 0:03:58 0:07:24 (4.4e-8, 2.5e-6, 1.1e-3) (20,5) (1771,230229) 0:28:14 0:30:30 0:43:27 (4.7e-7, 3.7e-6, 5.7e-6) (10,6) (220,5004) 0:00:11 0:00:14 0:00:20 (1.3e-7, 6.4e-7, 3.5e-2) (15,6) (680,38759) 0:03:14 0:04:19 0:04:53 (4.8e-8, 2.5e-3, 4.9e-2) (20,6) (1540,177099) 0:39:28 0:45:39 0:54:59 (2.8e-8, 6.6e-5, 1.0e-2) Computer Information: Matlab 7.10 on a Dell 64-bit Linux Desktop running CentOS (5.6) with 8GB memory and Intel(R) Core(TM) i7 CPU GHz. Li Wang (UCSD) SDP relaxations to POP and Application September 9, / 27

27 Thank you! Li Wang (UCSD) SDP relaxations to POP and Application September 9, / 27