Parallel implementation of primal-dual interior-point methods for semidefinite programs

Transcription

1 Parallel implementation of primal-dual interior-point methods for semidefinite programs Masakazu Kojima, Kazuhide Nakata Katsuki Fujisawa and Makoto Yamashita 3rd Annual McMaster Optimization Conference: Theory and Applications (MOPTA 03) July 30 - August 1, 2003 Hamilton, Ontario 1

2 1. SDP (semidefinite program) and its dual 2. Existing numerical methods for SDPs 3. Outline of our parallel implementation, SDPARA and SDPARA-C 4. Computation of search directions in SDPARA 5. Numerical results on SDPARA 6. Positive definite matrix Completion used in SDPARA-C 7. Numerical results on SDPARA-C 8. Conclusions 2

3 1. SDP (semidefinite program) and its dual 2. Existing numerical methods for SDPs 3. Outline of our parallel implementation, SDPARA and SDPARA-C 4. Computation of search directions in SDPARA 5. Numerical results on SDPARA 6. Positive definite matrix completion used in SDPARA-C 7. Numerical results on SDPARA-C 8. Conclusions 3

4 p=1 b py p sub.to X, S S n, y p R (1 p m) : variables A 0, A p S n, b p R (1 p m) : given data S n : the set of n n symmetric matrices n n U V = U ij V ij for every U, V R n n i=1 j=1 X O X S n X O X S n is positive semidefinite is positive definite 4

5 p=1 b py p sub.to Important features of SDPs n n matrix variables X, S S n, each of which involves n(n + 1)/2 real variables; for example, n = 2000 n(n + 1)/2 2 million. m linear equality constraints in P, where m can be large up to n(n + 1)/n. SDPs can be large scale easily Solving large scale SDPs (accurately) is a challenging subject. Special techniques for exploiting structure and sparsity Enormous computational power parallel computation 5

6 p=1 b py p sub.to Important features of SDPs n n matrix variables X, S S n, each of which involves n(n + 1)/2 real variables; for example, n = 2000 n(n + 1)/2 2 million. m linear equality constraints in P, where m can be large up to n(n + 1)/n. SDPs can be large scale easily Solving large scale SDPs (accurately) is a challenging subject. Special techniques for exploiting structure and sparsity Enormous computational power parallel computation 6

7 p=1 b py p sub.to 1. SDP (semidefinite program) and its dual 2. Existing numerical methods for SDPs 3. Outline of our parallel implementation, SDPARA and SDPARA-C 4. Computation of search directions in SDPARA 5. Numerical results on SDPARA 6. Positive definite matrix completion used in SDPARA-C 7. Numerical results on SDPARA-C 8. Conclusions 7

8 p=1 b py p sub.to Some existing numerical methods for SDPs IPMs (Interior-point methods) Primal-dual scaling, CSDP(Borchers), SDPA(Fujisawa-K-Nakata), SDPT3(Todd-Toh-Tutuncu), SeDuMi(F.Sturm) Dual scaling, DSDP(Benson-Ye) Nonlinear programming approaches Spectral bundle method(helmberg-kiwiel) Gradient-based log-barrier method(burer-monteiro-zhang) PENON(M. Kocvara) Generalized augmented Lagrangian method These methods are competing to each other. Large scale SDPs (e.g., n=10,000) and low accuracy Spectral bundle, Gradient-based log-barrier or IPMs using CG Medium scale SDPs (e.g. n, m = 1000) and high accuracy IPMs Parallel implementation of SDPA, DSDP, Spectral bundle method 8

9 p=1 b py p sub.to Some existing numerical methods for SDPs IPMs (Interior-point methods) Primal-dual scaling, CSDP(Borchers), SDPA(Fujisawa-K-Nakata), SDPT3(Todd-Toh-Tutuncu), SeDuMi(F.Sturm) Dual scaling, DSDP(Benson-Ye) Nonlinear programming approaches Spectral bundle method(helmberg-kiwiel) Gradient-based log-barrier method(burer-monteiro-zhang) PENON(M. Kocvara) Generalized augmented Lagrangian method These methods are competing to each other. Large scale SDPs (e.g., n=10,000) and low accuracy Spectral bundle, Gradient-based log-barrier or IPMs using CG Medium scale SDPs (e.g. n, m = 1000) and high accuracy IPMs Parallel implementation of SDPA, DSDP, Spectral bundle method 9

10 p=1 b py p sub.to Advantages of primal-dual IPMs (interior-point methods) Based on deep and strong theory (Nesterov-Nemirovskii) Highly accurate solutions. cf S.bundle and Gradient-based methods The number of iterations is small; usually iterations in practice, independent of sizes of SDPs. Disadvantage of primal-dual IPMs (interior-point methods) Heavy computation in each iteration Parallel execution of heavy computation in each iteration 10

11 p=1 b py p sub.to Advantages of primal-dual IPMs (interior-point methods) Based on deep and strong theory (Nesterov-Nemirovskii) Highly accurate solutions. cf S.bundle and Gradient-based methods The number of iterations is small; usually iterations in practice, independent of sizes of SDPs. Disadvantage of primal-dual IPMs (interior-point methods) Heavy computation in each iteration Parallel execution of heavy computation in each iteration 11

12 p=1 b py p sub.to 1. SDP (semidefinite program) and its dual 2. Existing numerical methods for SDPs 3. Outline of our parallel implementation, SDPARA and SDPARA-C 4. Computation of search directions in SDPARA 5. Numerical results on SDPARA 6. Positive definite matrix completion used in SDPARA-C 7. Numerical results on SDPARA-C 8. Conclusions 12

13 p=1 b py p sub.to Generic primal-dual IPM on a single CPU SDPA Step 0: Choose (X, y, S) = (X 0, y 0, S 0 ); X 0 O and S 0 O. Step 1: Compute a search direction (dx, dy, ds). Step 2: Choose α p and α d ; X + α p dx O and S + α d ds O. Let X = X + α p dx, (y, S) = (y, S) + α d (dy, ds). Step 3: Go to Step 1. Outline of our parallel implementation MPI (Message Passing Interface) for communication between CPUs. Myrinet-2000 between nodes, higher transmission than Gigabit Ethernet. ScaLAPACK (Scalable Linear Algebra PACKage) for parallel Cholesky factorization. 13

14 p=1 b py p sub.to Generic primal-dual IPM on a single CPU SDPA Step 0: Choose (X, y, S) = (X 0, y 0, S 0 ); X 0 O and S 0 O. Step 1: Compute a search direction (dx, dy, ds). matrix B Step 2: Choose α p and α d ; X + α p dx O and S + α d ds O. Let X = X + α p dx, (y, S) = (y, S) + α d (dy, ds). Step 3: Go to Step 1. parallel SDPA = = = = = = = SDPARA for large m 30, 000 Computation of Schur but small n 1, 000 complement matrix B Cholesky factorization of B Positive definite matrix completion technique SDPARA-C for large n 14

15 p=1 b py p sub.to Generic primal-dual IPM on a single CPU SDPA Step 0: Choose (X, y, S) = (X 0, y 0, S 0 ); X 0 O and S 0 O. Step 1: Compute a search direction (dx, dy, ds). Step 2: Choose α p and α d ; X + α p dx O and S + α d ds O. Let X = X + α p dx, (y, S) = (y, S) + α d (dy, ds). Step 3: Go to Step 1. parallel SDPA = = = = = = = SDPARA for large m 30, 000 Computation of Schur but small n 1, 000 complement matrix B Cholesky factorization of B Positive definite matrix completion technique SDPARA-C for larger n 15

16 p=1 b py p sub.to Generic primal-dual IPM on a single CPU SDPA Step 0: Choose (X, y, S) = (X 0, y 0, S 0 ); X 0 O and S 0 O. Step 1: Compute a search direction (dx, dy, ds). Step 2: Choose α p and α d ; X + α p dx O and S + α d ds O. Let X = X + α p dx, (y, S) = (y, S) + α d (dy, ds). Step 3: Go to Step 1. Outline of our parallel implementation MPI (Message Passing Interface) for communication between CPUs. Myrinet-2000 between nodes, higher transmission than Gigabit Ethernet. ScaLAPACK (Scalable Linear Algebra PACKage) for parallel Cholesky factorization. 16

17 p=1 b py p sub.to 1. SDP (semidefinite program) and its dual 2. Existing numerical methods for SDPs 3. Outline of our parallel implementation, SDPARA and SDPARA-C 4. Computation of search directions in SDPARA 5. Numerical results on SDPARA 6. Positive definite matrix completion used in SDPARA-C 7. Numerical results on SDPARA-C 8. Conclusions 17

18 p=1 b py p sub.to Computing HRVW/KSH/M search direction in SDPARA (X, y, S) ; X O and S O the current point (dx, dy, ds) HRVW/KSH/M search direction Bdy = r, where B S m ++, r Rm (X, y, S), A p, b p ds = D j=1 dy j, dx = (K XdS)S 1, dx = where D S n, K R n n (X, y, S), A p, b p ( dx + dx T ) /2 The computation of B requires O(m 2 n 2 ) arithmetic operations, and the solution of Bdy = r O(m 3 ) arithmetic operations SDPARA. The computation of dx = (K XdS)S 1 is expensive when n is large SDPARA-C. 18

19 p=1 b py p sub.to Computing HRVW/KSH/M search direction in SDPARA (X, y, S) ; X O and S O the current point (dx, dy, ds) HRVW/KSH/M search direction Bdy = r, where B S m ++, r Rm (X, y, S), A p, b p ds = D j=1 dy j, dx = (K XdS)S 1, dx = where D S n, K R n n (X, y, S), A p, b p ( dx + dx T ) /2 The computation of B requires O(m 2 n 2 ) arithmetic operations, and the solution of Bdy = r O(m 3 ) arithmetic operations SDPARA. The computation of dx = (K XdS)S 1 and dx = expensive when n is large SDPARA-C. ( dx + dx T ) /2 is 19

20 p=1 b py p sub.to Bdy = r, where B S m ++, B pq = Trace S 1 A p XA q (p, q = 1,..., m) 20

21 p=1 b py p sub.to Bdy = r, where B S m ++, B pq = Trace S 1 A p XA q (p, q = 1,..., m) In SDPARA: Compute each row of B by one cpu; S 1 A p X is common in pth row. Redistribute the elements of B in 2-dim. block-cyclic distribution. Apply the parallel Cholesky factorization to B for solving Bdy = r Processor 1 Processor 2 Processor 3 Processor 4 B 5 Processor 1 6 Processor 2 7 Processor Processor 4 Processor 1 Computation of 9 9 B by 4cpu s = dimensional block-cyclic dist. 21

22 p=1 b py p sub.to 1. SDP (semidefinite program) and its dual 2. Existing numerical methods for SDPs 3. Outline of our parallel implementation, SDPARA and SDPARA-C 6. Computation of search directions in SDPARA 5. Numerical results on SDPARA 6. Positive definite matrix completion used in SDPARA-C 7. Numerical results on SDPARA-C 8. Conclusions 22

23 p=1 b py p sub.to Numerical results on SDPARA PC cluster; 1.6GHz cpu and 768 MB memory in each node. Myrinet-2000 communication between the nodes. All data A p, b p are distributed to every node. Iterates {(X k, y k, S k )} are stored and updated in each nodes. Some heavy computations are done in parallel and their results are distributed to all nodes, but all other computations are done individually and independently in each node. Primal and dual feasibilities, relative duality gaps 1.0e 6 1.0e 7. 23

24 p=1 b py p sub.to Numerical results on SDPARA PC cluster; 1.6GHz cpu and 768 MB memory in each node. Myrinet-2000 communication between the nodes. All data A p, b p are distributed to every node. Iterates {(X k, y k, S k )} are stored and updated in each nodes. Some heavy computations are done in parallel and their results are distributed to all node, but all other computations are done individually and independently in each node. Primal and dual feasibilities, relative duality gaps 1.0e 6. 24

25 p=1 b py p sub.to Numerical results on SDPARA Test problem m >> size of A p (diag. blocks) small control11 from sdplib 1596 (100, 55) theta6 from sdplib thetag51 from sdplib sdp from quantum chemistry 1 15,313 (120,120,256) sdp from quantam chemistry 2 24,503 (153,153,324) Test probem m size of A p large maxg51 from sdplib qpg11 from sdplib qpg51 from sdplib torusg3-15 from dimacs 3,375 3,375 norm min

26 p=1 b py p sub.to Numerical results on SDPARA comp. time in sec Test probem m size of A p 1cpu 4cpu 16cpu 64cpu control (110,55) theta thetag M sdp from q.c 1 15,313 (120,120,256) M M sdp from q.c 2 24,503 (153,153,324) M M M: lack of memory Test probem m size of A p 1cpu 4cpu 16cpu 64cpu maxg qpg qpg M M M M torusg3-15 3,375 3,375 M M M M norm min M M M M 26

27 p=1 b py p sub.to Numerical results on SDPARA comp. time in sec Test probem m size of A p 1cpu 4cpu 16cpu 64cpu control (110,55) theta thetag M sdp from q.c 1 15,313 (120,120,256) M M sdp from q.c 2 24,503 (153,153,324) M M M: lack of memory Test probem m size of A p 1cpu 4cpu 16cpu 64cpu maxg qpg qpg M M M M torusg3-15 3,375 3,375 M M M M norm min M M M M 27

28 p=1 b py p sub.to 1. SDP (semidefinite program) and its dual 2. Existing numerical methods for SDPs 3. Outline of our parallel implementation, SDPARA and SDPARA-C 4. Computation of search directions in SDPARA 5. Numerical results on SDPARA 6. Positive definite matrix completion used in SDPARA-C 7. Numerical results on SDPARA-C 8. Conclusions 28

29 p=1 b py p sub.to Weak point in primal-dual interior-point methods The primal X is dense in general even when all A p s are sparse. The dual S = A 0 p=1 A py p inherits the sparsity of A p s. This difference causes a critical disadvantage of primal-dual interior-point methods compared to dual interior-point methods for large scale SDPs To overcome this disadvantage, we exploit E {(i, j) : [A p ] ij 0 for p} ( the aggregate sparsity pattern over all A p s) based on some fundamental results about positive matrix completion 29

30 p=1 b py p sub.to Weak point in primal-dual interior-point methods The primal X is dense in general even when all A p s are sparse. The dual S = A 0 p=1 A py p inherits the sparsity of A p s. This difference causes a critical disadvantage of primal-dual interior-point methods compared to dual interior-point methods for large scale SDPs To overcome this disadvantage, we exploit E {(i, j) : [A p ] ij 0 for p} ( the aggregate sparsity pattern over all A p s) based on some fundamental results about positive matrix completion. Fukuda-K-Murota-Nakata

31 p=1 b py p sub.to Example: m = 2, n = min sub.to X = 6, X 11 X 12 X 13 X 14 X 21 X 22 X 23 X 24 X 31 X 32 X 33 X 34 X 41 X 42 X 43 X Remember! C X = i,j X = 5, X O C ij X ij the aggregate sparsity pattern over all A p s E = {(i, j) in Red } X ij (i, j) E are unnecessary to evaluate the objective function and the equality constraints, but necessary for X O. Using positive definite matrix completion, we can generate each iterate (X, y, S) such that both X 1 and S have the same sparsity pattern as E when E is nicely sparse as in above example. In general, we need to expand E to a nicely sparse E. 31

32 p=1 b py p sub.to Example: m = 2, n = min sub.to X = 6, X 11 X 12 X 13 X 14 X 21 X 22 X 23 X 24 X 31 X 32 X 33 X 34 X 41 X 42 X 43 X Remember! C X = i,j X = 5, X O C ij X ij the aggregate sparsity pattern over all A p s E = {(i, j) in Red } X ij (i, j) E are unnecessary to evaluate the objective function and the equality constraints, but necessary for X O. Using positive definite matrix completion, we can generate each iterate (X, y, S) such that both X 1 and S have the same sparsity pattern as E when E is nicely sparse as in above example. In general, we need to expand E to a nicely sparse E. 32

33 p=1 b py p sub.to Technical details of the pd matrix completion E {(i, j) : [A p ] ij 0 for p}(the aggregate sparsity pattern) Each iteration of IPM, we need to (I) compute a step length α p such that X + α p dx O, (II) compute a search direction (dx, dy, ds). By applying the positive definite matrix completion technique, we can execute (I) and (II) using only X ij, (i, j) E for some nice and small E E (a chordal extension of E). Positive definite matrix completion problem: Given X ij = X ij, (i, j) E, assign values to the other elements X ij, (i, j) E so that X O. 33

34 (I) Computation of a step length α p : Example (m = 2, n = 4) min sub.to X = 6, X 11 X 12 X 13 X 14 X 21 X 22 X 23 X 24 X 31 X 32 X 33 X 34 X 41 X 42 X 43 X Remember! C X = i,j X = 5, X O C ij X ij ( ) Xii X Given X ij, (i, j) E, X ij, (i, j) E such that X O iff i4 X 4i X 44 O (i = 1, 2, 3) ( maximal principal submatrix in X ij s is pd.) Given X ij, dx ij (i, j) E, a step length α p is determined such that ( ) ( ) Xii X i4 dxii dx + α i4 X 4i X p O (i = 1, 2, 3) 44 dx 4i dx 44 X ij, dx ij (i, j) E are unnecessary! We can generalize this method to any aggregate sparsity pattern E characterized as a chordal graph. 34

35 7 7 example with the aggregate sparsity pattern E = {(i, j) : A ij 0}, A = where : a nonzero element. E is represented by the graph: Let E be a chordal extension of E, i.e., E such that minimal cycle has no more than 3 edges or the aggregate sparsity pattern E induced by the symbolic Cholesky factorization of A. E = E {(4, 5)} : a chordal extension of E 35

36 7 7 example with the aggregate sparsity pattern E = {(i, j) : A ij 0}, A = where : a nonzero element. E is represented by the graph: Let E be a chordal extension of E, i.e., E E such that minimal cycle has no more than 3 edges or the aggregate sparsity pattern E induced by the symbolic Cholesky factorization of A. E = E {(4, 5)} : a chordal extension of E 36

37 E : A = Theorem (Gron-Sá et.al. 84) Given X ij = X ij, (i, j) E, X ij, (i, j) E ; X O iff X CC O for maximal clique C of E In the example above, the maximal cliques are C 1 = {1, 2, 3}, C 2 = {3, 4, 5}, C 3 = {4, 5, 7}, C 4 = {6, 7} Hence α p is chosen; X CC + α p dx CC O for maximal clique C of E, instead of X + α p dx O. This requires less cpu time and memory. We need only X ij = X ij, dx ij = dx ij, (i, j) E. 37

38 (II) Computation of a search direction (dx, dy, ds). Let ˆX S n ++ be the matrix that maximizes its determinant among all pd completions of a given partial matrix with X ij = X ij, (i, j) E. Then ˆX 1 and the Cholesky factorization LL T of ˆX 1, which we can easily compute, enjoys the same sparsity pattern as E. Store the Cholesky factorization LL T of ˆX 1 instead of ˆX itself, and the Cholesky factorization NN T of S. Then, all A p s, N and L enjoy the same sparsity pattern as E. Use L T L 1 and NN T instead of ˆX and S, respectively. In particular, B pq Trace A p ˆXA q S 1 = Trace A p L T L 1 A q N T N 1. for computing the m m coefficient matrix B of the Schur complement equation Bdy = r for (dx, dy, ds). Considerable savings in memory. Suitable for parallel computation to reduce computational time. 38

39 (II) Computation of a search direction (dx, dy, ds). Let ˆX S n ++ be the matrix that maximizes its determinant among all pd completions of a given partial matrix with X ij = X ij, (i, j) E. Then ˆX 1 and the Cholesky factorization LL T of ˆX 1, which we can easily compute, enjoys the same sparsity pattern as E. Store the Cholesky factorization LL T of ˆX 1 instead of ˆX itself, and the Cholesky factorization NN T of S. Then, all A p s, N and L enjoy the same sparsity pattern as E. Use L T L 1 and NN T instead of ˆX and S, respectively. In particular, B pq Trace A p ˆXA q S 1 = Trace A p L T L 1 A q N T N 1. for computing the m m coefficient matrix B of the Schur complement equation Bdy = r for (dx, dy, ds). Considerable savings in memory. Suitable for parallel computation to reduce computational time. 39

40 (II) Computation of a search direction (dx, dy, ds). Let ˆX S n ++ be the matrix that maximizes its determinant among all pd completions of a given partial matrix with X ij = X ij, (i, j) E. Then ˆX 1 and the Cholesky factorization LL T of ˆX 1, which we can easily compute, enjoys the same sparsity pattern as E. Store the Cholesky factorization LL T of ˆX 1 instead of ˆX itself, and the Cholesky factorization NN T of S. Then, all A p s, N and L enjoy the same sparsity pattern as E. Use L T L 1 and NN T instead of ˆX and S, respectively. In particular, B pq Trace A p ˆXA q S 1 = Trace A p L T L 1 A q N T N 1. for computing the m m coefficient matrix B of the Schur complement equation Bdy = r for (dx, dy, ds). Considerable savings in memory. Suitable for parallel computation to reduce computational time. 40

41 p=1 b py p sub.to 1. SDP (semidefinite program) and its dual 2. Existing numerical methods for SDPs 3. Outline of our parallel implementation, SDPARA and SDPARA-C 4. Computation of search directions in SDPARA 5. Numerical results on SDPARA 6. Positive definite matrix completion used in SDPARA-C 7. Numerical results on SDPARA-C 8. Conclusions 41

42 p=1 b py p sub.to 7. Numerical results on SDPARA-C Test problem m size of A p 1cpu 4cpu 16cpu 64cpu theta thetag M M: lack of memory Test problem m size of A p 1cpu 4cpu 16cpu 64cpu maxg qpg qpg torusg3-15 3,375 3, norm min

43 p=1 b py p sub.to Comparison of Real Computational Time (sec) between SDPARAC, SDPARA and PDSDP (Benson). control10, sdplib, m=1326, n=(100,50) #CPU SDPARA-C SDPARA PDSDP theta6, sdplib, m=4375, n=300 #CPU SDPARA-C SDPARA PDSDP

44 p=1 b py p sub.to maxg51, sdplib, max cut m=1000, n=1000 #CPU SDPARA-C SDPARA PDSDP qpg51, sdplib, m=1000, n= M(970MB) M M M 495 torusg3-15 dimacs max cut, m=3,375, n=3, M(920MB) M M M

45 p=1 b py p sub.to 1. SDP (semidefinite program) and its dual 2. Existing numerical methods for SDPs 3. Outline of our parallel implementation, SDPARA and SDPARA-C 4. Computation of search directions in SDPARA 5. Numerical results on SDPARA 6. Positive definite matrix completion used in SDPARA-C 7. Numerical results on SDPARA-C 8. Conclusions 45

46 p=1 b py p sub.to Two types of parallel primal-dual interior-point methods for SDPs (a) Parallel implementation SDPARA of SDPA, suitable for large m and smaller n. The largest problem solved is m size of A p 1cpu 4cpu 16cpu 64cpu sdp from q.c 2 24,503 (153,153,324) M M (b) SDPARA-C = SDPARA + the pd matrix completion, suitable for larger n. The largest problems solved are m size of A p 1cpu 4cpu 16cpu 64cpu torusg3-15 3,375 3, norm min