FINDING PARALLELISM IN GENERAL-PURPOSE LINEAR PROGRAMMING

Transcription

1 FINDING PARALLELISM IN GENERAL-PURPOSE LINEAR PROGRAMMING Daniel Thuerck 1,2 (advisors Michael Goesele 1,2 and Marc Pfetsch 1 ) Maxim Naumov 3 1 Graduate School of Computational Engineering, TU Darmstadt 2 Graphics, Capture and Massively Parallel Computing, TU Darmstadt 3 NVIDIA Research TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 1

2 INTRODUCTION TO LINEAR PROGRAMMING TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 2

3 Linear Programs min c x s.t. Ax b x 0 Linear objective function Linear constraints c where A = a 1 T a m T and b = b 1 b m TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 3

4 Linear Programs: Applications [3P Logistics] TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 4

5 Lower-Level Parallelism in LP INTERNALS OF AN LP SOLVER TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 5

6 Solving LPs min c x s.t. Ax = b. x 0 A is m n matrix, with m n A is sparse and has full row-rank Variables may be bounded: l x u Standard LP format TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 6

7 Solving LPs Simplex Interior Point c c TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 7

8 Solving LPs Simplex Interior Point (IPM) Basis (active set) Augmented (Newton) System D A A A B = Normal Equations AD 1 A TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 8

9 Solving LPs IPM / Aug. System IPM / Normal Equations D A A AD 1 A (m + n) (m + n), sparse Symmetric, indefinite Solution: Indefinite LDL T or MINRES method m m, SPD, might be dense Squared condition number Solution: Cholesky-factorization or CG method TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 9

10 Solving LPs IPM / Aug. System IPM / Normal Equations D A A AD 1 A (m + n) (m + n), sparse Symmetric, indefinite Solution: Indefinite LDL T or MINRES method m m, SPD, might be dense Squared condition number Solution: Cholesky-factorization or CG method TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 10

11 Introducing culip-lp An ongoing implementation of Mehrotra s Primal-Dual interior point algorithm [1], featuring... (Iterative) Linear Algebra based on the Augmented Matrix approach, Full-rank guarantees, Comprehensive preprocessing & prescaling. Towards solving large-scale LPs on the GPU as open source for everybody TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 11

12 Progress report IMPLEMENTING CULIP-LP TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 12

13 Solver architecture Preprocess Scale Standardize IPM loop TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 13

14 Solver architecture Preprocess Scale Standardize IPM loop Input data: Constraints Constraints Objective vector Bounds (on some variables) A eq x = b eq A le x b le c l, u TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 14

15 Solver architecture Preprocess Scale Standardize IPM loop A eq x = b eq A le x b le c l, u Storage format: CSR Compressed sparse row format Provides efficient row-based access by 3 arrays: row_ptr a e b c d col_ind a b c d e val TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 15

16 Solver architecture Preprocess Scale Standardize IPM loop A eq x = b eq A le x b le c l, u Example: LP pb-simp-nonunif (see [2]) Input matrix: 1,4 Mio x 23k with 4,36 Mio nonzeros Removed 1 singleton inequality Removed 3629 low-forcing constraints Removed 1 fixed variable Removed 1,1 Mio (!) singleton inequalities Result: approx. 3,6 Mio nonzeros removed Execute in rounds TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 16

17 Solver architecture Preprocess Scale Standardize IPM loop A eq x = b eq A le x b le c l, u Goal: Reduce element variance in matrices Scaling [3] makes a difference 1. Geometric scaling (1x 4x) A i, = 2. Equilibration (1x) A i, max A i, min( A i, ) A i, = A i, A i, TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 17

18 Solver architecture Preprocess Scale Standardize IPM loop Goal: Format LP in standard form Shift variables: l x u 0 x u + l Split (free) variables x x = x + x x +, x 0 Build std matrix: A le I = b Le A eq b eq A eq x = b eq A le x b le c l, u TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 18

19 Solver architecture Preprocess Scale Standardize IPM loop Ensure A has full rank (symbolically) PAQ = m u Ax = b c u m c TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 19

20 Solver architecture Preprocess Scale Standardize IPM loop Goal: Solve KKT conditions by Newton steps Steps: Augmented matrix assembly Solving the (indefinite) augmented matrix Solve twice: predictor and corrector Stepsize along v = v p + v c v p v c Ax = b c u TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 20

21 Solving the augmented system Iterative strategy: Symmetric, indefinite: use MINRES [4] (in parts) D A A Equilibrate system implicitly Preconditioner: Experiments ongoing Direct strategy: Symmetric, indefinite: use SPRAL SSIDS [5] Reordering by METIS [6] Scaling for large pivots TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 21

22 Intermediate findings PERFORMANCE EVALUATION TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 22

23 Benchmark problems Problem name [7] M N NNZ ex9 40,962 10, ,112 ex10 696,608 17,680 1,162,000 neos , , ,166 bley_xl1 175, ,391 map06 328, , ,920 map10 328, , ,920 nb10tb 150, ,172,289 neos , ,040 1,855,220 in 1,526,202 1,449,074 6,811, TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 23

24 Performance Problem name [7] NNZ CLP barr [sec] culip-lp [sec] ex9 517,112 X (NC) 81 ex10 1,162,000 X (NS) 141 neos , bley_xl1 869,391 X (NS) 1,492 map06 549,920 X (NC) 466 map10 549,920 X (NC) 615 nb10tb 1,172,289 X (NC) 2,461 neos ,855, in 6,811,639 X (NS) NC X failed, NS did not start 1 st iteration, NC did not converged within 1 hour TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 24

25 time [sec] Runtime breakdown MINRES Predictor Corrector SPRAL IPM step Example: map10 [7] Problem: map10 [7] TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 25

26 Iterations Relative Residual Iterative vs. direct methods 6000 MINRES Iterations 1.E+00 MINRES relative residual E E Predictor Corrector 1.E-03 1.E-04 1.E-05 1.E IPM step 1.E-07 Example: map10 [7] IPM step TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 26

27 Numerical difficulty Condition of matrix D A A depends mainly on D = diag(x) diag(s) Remedies with strong duality towards the end often yielding 2x2 pivoting in factorizations (e.g. LDL in SPRAL) Preconditioning for MINRES or GMRES max(x i s i ) min x i s i where x T =[x 1,,x n ] are solution and s T =[s 1,,s n ] are slack variables TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 27

28 Findings on the solver s performance We know that individual components attain speedup on the GPU Linear algebra components, such as preconditioned MINRES, GMRES, etc. Graph reorderings and matchings LP problems: Medium-sized problems: faster than open-source IPM code CLP We can solve many large problems where CLP fails However, more time is needed to integrate all components together on the GPU TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 28

29 What s keeping you from optimizing your runtime? LP Solver (a.k.a the black box ) = Rank estimation Preconditioner MPS I/O Preprocessing Graph partitioning Bipartite matching Krylov-Solver Component BFS Matrix factorization Max-flow SpMV Equilibration Matrix reordering LP scaling TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 29

30 Future Work Numerics Preconditioning techniques Investigate influence of 2x2 pivots in LDL T factorization Tuning Benchmark & optimize warp-based kernels in preprocessing Never explicitly form the standardized matrix TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 30

31 Higher-Level Parallelism in LP FEASIBILITY STUDY: LP DECOMPOSITIONS TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 31

32 Solving an LP: The usual setup Large LP Solution Preprocess LP Solver Standardize (a.k.a the black box ) Solve TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 32

33 Higher-level parallelism by LP decomposition Slave LP 1 Large LP Apply LP decomposition Master LP Slave LP k Solution Assemble master/slave solutions TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 33

34 LP-decompositions: feasibility Decomposition works on structure of the constraint matrix A: Benders [9] Dantzig-Wolfe [10] TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 34

35 LP-decompositions: prototype Implemented a Benders decomposition using hypergraph partitioning: TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 35

36 Acknowledgements The work of Daniel Thuerck is supported by the 'Excellence Initiative' of the German Federal and State Governments and the Graduate School of Computational Engineering at Technische Universität Darmstadt TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 36

37 References [1] Mehrotra, Sanjay. "On the implementation of a primal-dual interior point method." SIAM Journal on optimization 2.4 (1992): [2] Gondzio, Jacek. "Presolve analysis of linear programs prior to applying an interior point method."informs Journal on Computing 9.1 (1997): [3] Gondzio, Jacek. "Presolve analysis of linear programs prior to applying an interior point method."informs Journal on Computing 9.1 (1997): [4] Paige, Christopher C., and Michael A. Saunders. "Solution of sparse indefinite systems of linear equations." SIAM journal on numerical analysis 12.4 (1975): [5] Paige, Christopher C., and Michael A. Saunders. "Solution of sparse indefinite systems of linear equations." SIAM journal on numerical analysis 12.4 (1975): TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 37

38 References [6] Karypis, George, and Vipin Kumar. "A fast and high quality multilevel scheme for partitioning irregular graphs." SIAM Journal on scientific Computing 20.1 (1998): [7] Koch, Thorsten, et al. "MIPLIB 2010." Mathematical Programming Computation 3.2 (2011): [8] Forrest, John, David de la Nuez, and Robin Lougee-Heimer. "CLP user guide." IBM Research (2004). [9] Benders, Jacques F. "Partitioning procedures for solving mixed-variables programming problems."numerische mathematik 4.1 (1962): [10] Dantzig, George B., and Philip Wolfe. "Decomposition principle for linear programs." Operations research 8.1 (1960): TU Darmstadt GCC / GSC CE Daniel Thuerck, Maxim Naumov 38