Mechanizing Optimization and Statistics

Transcription

1 Mechanizing Optimization and Statistics Ashish Agarwal Yale University IBM Programming Languages Day Watson Research Center July 29, 2010 Ashish Agarwal () 1 / 34

2 Acknowledgments Optimization: Ignacio Grossmann (Carnegie Mellon) Nick Sawaya and Vikas Goel (Exxon Mobil) Indexing: Bob Harper (Carnegie Mellon) Statistics: Sooraj Bhat and Alex Gray (GeorgiaTech) Rich Vuduc (GeorgiaTech, linear algebra and autotuning) Ashish Agarwal () 2 / 34

3 Linear programs (LP) Dantzig (1982) x x 1 x x x 1 + x Ashish Agarwal () 3 / 34

4 Linear programs (LP) Dantzig (1982) x x 1 max x 1 x 2 x x x 1 + x Ashish Agarwal () 3 / 34

5 Linear programs (LP) Dantzig (1982) x x 1 max x 1 x 2 x x x 1 + x Cannot represent multiple polyhedra. Ashish Agarwal () 3 / 34

6 Declaring discrete choice with disjunction Disjunctive programs (DP), Balas (1974), Jeroslow and Lowe (1984), Raman and Grossmann (1994) x R 1 R x 1 x 1 1 x 2 1 } x 1 + x 2 5 {{ } R 1 [ ] 5 x1 8 4 x 2 7 }{{} R 2 Ashish Agarwal () 4 / 34

7 Declaring discrete choice with disjunction Disjunctive programs (DP), Balas (1974), Jeroslow and Lowe (1984), Raman and Grossmann (1994) x R 1 R x 1 x 1 1 x 2 1 x 1 + x 2 5 } {{ } R 1 [ 5 x1 8 4 x 2 7 ] } {{ } R 2 Language of DP extends LP with disjunction Ashish Agarwal () 4 / 34

8 Declaring discrete choice with disjunction Disjunctive programs (DP), Balas (1974), Jeroslow and Lowe (1984), Raman and Grossmann (1994) x R 1 R x 1 x 1 1 x 2 1 x 1 + x 2 5 } {{ } R 1 [ 5 x1 8 4 x 2 7 ] } {{ } R 2 Language of DP extends LP with disjunction Few algorithms for solving DPs directly. Ashish Agarwal () 4 / 34

9 Declaring discrete choice with integers Mixed-integer linear programs (MILP) Basic idea: multiply terms by y {0, 1} 0 y 1 x 3.0y + 2.0(1 y) if y = 1, then x 3.0 if y = 0, then x 2.0 Ashish Agarwal () 5 / 34

10 Declaring discrete choice with integers Mixed-integer linear programs (MILP) Basic idea: multiply terms by y {0, 1} 0 y 1 x 3.0y + 2.0(1 y) if y = 1, then x 3.0 if y = 0, then x 2.0 Language of MILP extends LP with the integer type Ashish Agarwal () 5 / 34

11 Declaring discrete choice with integers Mixed-integer linear programs (MILP) Basic idea: multiply terms by y {0, 1} 0 y 1 x 3.0y + 2.0(1 y) if y = 1, then x 3.0 if y = 0, then x 2.0 Language of MILP extends LP with the integer type Goal: Express as DP (intuitive) Convert to MILP (accepted by solvers) Ashish Agarwal () 5 / 34

12 Multiple Transformation Techniques Available Choice affects computational efficiency x 2 best big-m reformulation x 2 convex-hull reformulation x x 1 Ashish Agarwal () 6 / 34

13 System overview disjunctive constraint [A 1 x b 1 ] [A 2 x b 2 ] x2 7 6 R R x Boolean propositions big-m convex-hull indicator constr x2 x x x y = 0 A 1 x b 1 y = 1 A 2 x b 2 Pure integer inequalities Ashish Agarwal () 7 / 34

14 Previous definition Convex-hull reformulation of is [A 1 x b 1 ] [A 2 x b 2 ] A 1 x 1 b 1 y 1 A 2 x 2 b 2 y 2 y 1 + y 2 = 1 x = x 1 + x 2 Ashish Agarwal () 8 / 34

15 Previous definition Convex-hull reformulation of is [A 1 x b 1 ] [A 2 x b 2 ] A 1 x 1 b 1 y 1 A 2 x 2 b 2 y 2 y 1 + y 2 = 1 x = x 1 + x 2 Insufficient for automation Real programs not declared in canonical matrix form How are variables introduced? Disjuncts should be bounded, how is this checked? How are variable bounds tracked? Disaggregated variables should have same bounds as those they replace (except range must include zero) Ashish Agarwal () 8 / 34

16 A syntactic foundation for mathematical programs Agarwal, Bhat, Gray, Grossmann (2010) ρ ::= [r L,r U ] [r L, ) (,r U ] real r L,r U r L, ) (,r U int {true} {false} bool e ::= x r true false note e 1 or e 2 e 1 ande 2 e e 1 + e 2 e 1 e 2 e 1 e 2 c ::= T F istrue e e 1 = e 2 e 1 e 2 c 1 c 2 c 1 c 2 x :ρ c p ::= max x1 :ρ 1,...,x m:ρ m {e c} Υ ::= Υ,x :ρ Ashish Agarwal () 9 / 34

17 Convex-hull transformation Agarwal, Bhat, Gray, Grossmann (2010) Υ c cvx c { } prop Υ c j c j { Υ c j x j Υ c A c B cvx Υ j {A,B} } { y j { x j / x } } c j c j j {A,B} j {A,B} ( x A : ρ ( x B : ρ y A : 0,1 y B : 0,1 x = x A + x B) ( y A + y B = 1 ) (c A c B ) c 1,...,xj m j Compile disjuncts and context Add bounding constraints Υ ctxt Substitute disaggregated variables in each disjunct Multiply constant terms by respective y ) Ashish Agarwal () 10 / 34

18 Example: single disjunctive constraint Input var x:<10.0, 100.0> var w:<2.0, 50.0> min x + w subject_to (x <= w) disj (x >= w + 4.0) Output var x:<10.0, 100.0> var w:<2.0, 50.0> min x + w subject_to exists y1:[0, 1] exists y2:[0, 1] exists x1:<0.0, 100.0> exists x2:<0.0, 100.0> exists w1:<0.0, 50.0> exists w2:<0.0, 50.0> w = w1 + w2, x = x1 + x2, y1 + y2 = 1, 10.0 * y1 <= x1, x1 <= * y1, 2.0 * y1 <= w1, w1 <= 50.0 * y1, x1 <= w1, 10.0 * y2 <= x2, x2 <= * y2, 2.0 * y2 <= w2, w2 <= 50.0 * y2, x2 >= w * y2 Ashish Agarwal () 11 / 34

19 Example: single disjunctive constraint Input var x:<10.0, 100.0> var w:<2.0, 50.0> min x + w subject_to (x <= w) disj (x >= w + 4.0) Output generated in MPS and AMPL formats Implemented as a DSL embedded in OCaml Output var x:<10.0, 100.0> var w:<2.0, 50.0> min x + w subject_to exists y1:[0, 1] exists y2:[0, 1] exists x1:<0.0, 100.0> exists x2:<0.0, 100.0> exists w1:<0.0, 50.0> exists w2:<0.0, 50.0> w = w1 + w2, x = x1 + x2, y1 + y2 = 1, 10.0 * y1 <= x1, x1 <= * y1, 2.0 * y1 <= w1, w1 <= 50.0 * y1, x1 <= w1, 10.0 * y2 <= x2, x2 <= * y2, 2.0 * y2 <= w2, w2 <= 50.0 * y2, x2 >= w * y2 Ashish Agarwal () 11 / 34

20 Switched flow process on off m α m β hi low α M max β M min F out Ashish Agarwal () 12 / 34

21 Switched flow process, example constraint Pump α has three kinds of mode transition dynamics: i N\{n}, istrue Z α (on,off,i) ĉ α (i) = 0.0 ˆr α (i) = R e (i) istrue Z α (off,on,i) R e (i) 2.0 ĉ α (i) = 50.0 ˆr α (i) = R e (i) istrue YY α (i) ĉ α (i) = 0.0 ˆr α (i) = 0.0 Booleans and disjunction enable the natural modeling of such logical relations between constraints. Ashish Agarwal () 13 / 34

22 Switched flow process, example constraint Pump α has three kinds of mode transition dynamics: i N\{n}, istrue Z α (on,off,i) ĉ α (i) = 0.0 ˆr α (i) = R e (i) istrue Z α (off,on,i) R e (i) 2.0 ĉ α (i) = 50.0 ˆr α (i) = R e (i) istrue YY α (i) ĉ α (i) = 0.0 ˆr α (i) = which interact with each other: i N\{n}, a {α,β}, istrue YY a (i) q Q a Z a (q,q,i) Booleans and disjunction enable the natural modeling of such logical relations between constraints. Ashish Agarwal () 13 / 34

23 Switched flow process, comparison to ILOG Concert Method #vars (#binary) #constr. (#IC) time (sec) flow-concert 1061 (874) 1080 (718) flow-ic 477 (291) 1001 (438) flow-bm 477 (291) flow-ch 1194 (631) Ashish Agarwal () 14 / 34

24 Switched flow process, comparison to ILOG Concert Method #vars (#binary) #constr. (#IC) time (sec) flow-concert 1061 (874) 1080 (718) flow-ic 477 (291) 1001 (438) flow-bm 477 (291) flow-ch 1194 (631) All 3 of our methods improve on state-of-the-art. Ashish Agarwal () 14 / 34

25 Strip packing Ashish Agarwal () 15 / 34

26 Strip packing, formulation The most natural formulation uses disjunction. min no overlapping stay in bounds length s.t. length x i + L i i N [x i + L i x j ] [x j + L j x i ] [y i H i y j ] [y j H j y i ] i,j N,i < j { 0 xi L max L i i N H i y i W i N (x i,y i ) is the position of the top-left corner of rectangle i. Ashish Agarwal () 16 / 34

27 Strip packing, comparison to expert Method #vars (#binary) #constr. (#IC) time (sec) pack12-ic 289 (264) 342 (264) 1.83 pack12-bm 289 (264) pack12-ch 1345 (264) pack12-bm-expert 289 (264) pack12-ch-expert 1345 (264) pack21-ic 883 (840) 1071 (840) pack21-bm 883 (840) pack21-ch 4243 (840) pack21-bm-expert 883 (840) pack21-ch-expert 4243 (840) 5271 > Ashish Agarwal () 17 / 34

28 Strip packing, comparison to expert Method #vars (#binary) #constr. (#IC) time (sec) pack12-ic 289 (264) 342 (264) 1.83 pack12-bm 289 (264) pack12-ch 1345 (264) pack12-bm-expert 289 (264) pack12-ch-expert 1345 (264) pack21-ic 883 (840) 1071 (840) pack21-bm 883 (840) pack21-ch 4243 (840) pack21-bm-expert 883 (840) pack21-ch-expert 4243 (840) 5271 > Our mechanizations perform just as well as expert encodings. Ashish Agarwal () 17 / 34

29 Indexing is Ubiquitous We solved a problem with 150,000 equations and 25,000 variables. How were so many equations and variables declared? Ashish Agarwal () 18 / 34

30 Indexing is Ubiquitous We solved a problem with 150,000 equations and 25,000 variables. How were so many equations and variables declared? Sometimes, use matrix notation. Often, use indexing. Ashish Agarwal () 18 / 34

31 Indexing is Ubiquitous We solved a problem with 150,000 equations and 25,000 variables. How were so many equations and variables declared? Sometimes, use matrix notation. Often, use indexing. Indexed operators: Families of equations: n i=1 x i i [1... n] x i+1 = x i + y i Indexing is a meta-programming feature Index variables i distinct from mathematical variables x Ashish Agarwal () 18 / 34

32 Complex Index Sets Arise in Real Problems Job shop scheduling: j J s S j j Pre j,s t j,s t j,s Mappings from sets to set of all sets: S Dependent types: S j depends on value of j Ashish Agarwal () 19 / 34

33 Indexing Language: Syntax Agarwal (2006) Index Expressions Index Sets (Types) ε ::= i k (ε 1,...,ε m ) ε.k ε ε 1 + ε 2 ε 1 ε 2 ε 1 ε 2 case ε of {k j ε j } m j=1 σ ::= [ε L..ε U ] i 1 : σ 1 i m : σ m case ε of {k j σ j } m j=1 λi σ σ ε σ :: κ Kinds κ ::= IndexSet i : σ κ Ashish Agarwal () 20 / 34

34 Example Index Sets set JOBS = { a, b, c } set STAGES = fn i. case i of a => { s1, s2 } b => { s1, s3, s4 } c => { s3, s4 } set JOBS_STAGES = i:jobs * STAGES[i] Explicitly: {( a, s1 ), ( a, s2 ), ( b, s1 ), ( b, s3 ), ( b, s4 ), ( c, s3 ), ( c, s4 )} Ashish Agarwal () 21 / 34

35 Memory Reduction Load this program: Other software expand this to: i [1... n] x i+1 = x i + y i x 2 = x 1 + y 1 x 3 = x 2 + y 2 x 4 = x 3 + y 3 x 5 = x 4 + y 4... Ashish Agarwal () 22 / 34

36 Memory Reduction Load this program: Other software expand this to: i [1... n] x i+1 = x i + y i x 2 = x 1 + y 1 x 3 = x 2 + y 2 x 4 = x 3 + y 3 x 5 = x 4 + y 4... We retain indexing structure: Memory requirements reduced from O(n) to O(1). Ashish Agarwal () 22 / 34

37 Computational Improvements Input to our software: i:[1..10] w x i Our software s output: i:[1..10] 10.0 y i w i w i 90.0 y i [ 5.0 yi x i,d ] x i,d 75.0 y i w i x i,i y i d:[1..10] Ashish Agarwal () 23 / 34

38 Computational Improvements Input to our software: i:[1..10] w x i Our software s output: i:[1..10] 10.0 y i w i w i 90.0 y i [ 5.0 yi x i,d ] x i,d 75.0 y i w i x i,i y i d:[1..10] Reformulation time reduced from O(n) to O(1). Ashish Agarwal () 23 / 34

39 Indexing Language: Type System and Semantics Agarwal (2006) Usual Way: Syntax Type System Semantics Existence is prior to meaning. Alternative Way: Syntax Semantics Type System Meaning is prior to existence. Ashish Agarwal () 24 / 34

40 Indexing Language: Type System and Semantics Agarwal (2006) Usual Way: Syntax Type System Semantics Existence is prior to meaning. Alternative Way: Syntax Semantics Type System Meaning is prior to existence. Admits more programs. Possible only because all types are finitary. Ashish Agarwal () 24 / 34

41 What Are Random Variables? Wasserman (2004) says: A random variable is a mapping X : Ω R that assigns a real number X(ω) to each outcome ω. Ashish Agarwal () 25 / 34

42 What Are Random Variables? Wasserman (2004) says: However: A random variable is a mapping X : Ω R that assigns a real number X(ω) to each outcome ω. Treated as real: P(X 5) Not random: We write X Bernoulli(p) to mean that X is exactly distributed as f (x) = p x (1 p) 1 x for x {0,1} Ashish Agarwal () 25 / 34

43 What Are Random Variables? Not variables: Cannot substitute occurrences of X for anything. e.g. In P(X 5), certainly cannot replace X with its distribution. Dependence matters. e.g. Two random variables X and Y, both distributed as Bernoulli(0.5), each 0 or 1 with probability 0.5. What is P(X + Y = 2)? Perhaps 0.25? But not if Y = 1 X. Ashish Agarwal () 26 / 34

44 What Are Random Variables? Not variables: Cannot substitute occurrences of X for anything. e.g. In P(X 5), certainly cannot replace X with its distribution. Dependence matters. e.g. Two random variables X and Y, both distributed as Bernoulli(0.5), each 0 or 1 with probability 0.5. What is P(X + Y = 2)? Perhaps 0.25? But not if Y = 1 X. Random variables are neither random nor variable. Ashish Agarwal () 26 / 34

45 Previous Work Giry (1981), Jones and Plotkin (1989) Probability distributions are a monad. Kozen (1981) Formalized semantics. Ramsey and Pfeffer (2002) Efficient expectations, but discrete distributions only. Park, Pfenning, and Thrun (2004) Continuous distributions also, but support only sampling. Erwig and Kollmansberger (2006) Provide Haskell library, but discrete distributions only, computational efficiency not optimized. Ashish Agarwal () 27 / 34

46 Previous Work Giry (1981), Jones and Plotkin (1989) Probability distributions are a monad. Kozen (1981) Formalized semantics. Ramsey and Pfeffer (2002) Efficient expectations, but discrete distributions only. Park, Pfenning, and Thrun (2004) Continuous distributions also, but support only sampling. Erwig and Kollmansberger (2006) Provide Haskell library, but discrete distributions only, computational efficiency not optimized. Our goal: Unify these results in a single system. Ashish Agarwal () 27 / 34

47 Syntax: Probability Language Bhat, Agarwal, Gray, Vuduc (2010) T ::= Bool Int Real T 1 T 2 Prob T E ::= x true false r E 1 + E 2 E 1 E 2 (E 1,E 2 ) fst E snd E if E 1 then E 2 else E 3 E 1 = E 2 E 1 E 2 uniform return E let x E 1 in E 2 Ashish Agarwal () 28 / 34

48 Language: Type System Example typing rule: Γ E 1 : Prob T 1 Γ,x : T 1 E 2 : Prob T 2 Γ let x E 1 in E 2 : Prob T 2 Ashish Agarwal () 29 / 34

49 Language: Type System Example typing rule: Γ E 1 : Prob T 1 Γ,x : T 1 E 2 : Prob T 2 Γ let x E 1 in E 2 : Prob T 2 Pass: var U uniform in return (U 0.7) Fail: var U uniform in (U 0.7) Ashish Agarwal () 29 / 34

50 Gaussian Model Height (cm) Ashish Agarwal () 30 / 34

51 Mixture of Gaussians Model Height (cm) Ashish Agarwal () 31 / 34

52 Trying alternative statistical models Formulation: X i Normal(θ,1) ˆθ = arg max f (x θ) θ Ashish Agarwal () 32 / 34

53 Trying alternative statistical models Formulation: X i Normal(θ,1) ˆθ = arg max f (x θ) θ Solution: n ˆθ = 1 n i=1 x i Ashish Agarwal () 32 / 34

54 Trying alternative statistical models Formulation: X i Normal(θ,1) ˆθ = arg max f (x θ) θ Formulation: Z i Bernoulli(0.5) X i Normal((1 Z i )θ 0 + Z i θ 1,1) ˆθ = arg max f (x θ) θ Solution: n ˆθ = 1 n i=1 x i Ashish Agarwal () 32 / 34

55 Trying alternative statistical models Formulation: X i Normal(θ,1) ˆθ = arg max f (x θ) θ Formulation: Z i Bernoulli(0.5) X i Normal((1 Z i )θ 0 + Z i θ 1,1) ˆθ = arg max f (x θ) θ Solution: ˆθ = 1 n n i=1 x i Solution: (ˆθ 0, ˆθ 1 ) := rand(); while (...) for i = 1 to n do γ i := φ(x i ; ˆθ 1,1)/(φ(x i ; ˆθ 0,1)+φ(x i ; ˆθ 1,1)); ˆθ 0 := P n i=1 (1-γ i)*x i / P n i=1 (1-γ i); ˆθ 1 := P n i=1 γ i*x i / P n i=1 γ i; return (ˆθ 0, ˆθ 1 ); Ashish Agarwal () 32 / 34

56 Interactive algorithm assistant Bhat, Agarwal, Gray, Vuduc (2010) Features Status enter problems apply schemas undo/redo combinators can solve several textbook examples of MLE, incl. via EM autotuning + more sophisticated code generation is planned Ashish Agarwal () 33 / 34

57 Conclusions Automated bigm and convex-hull methods Beginnings of a formalization of probability and statistics Library of transformations Formalization of indexing provides: advances on previous MP languages: e.g. GAMS, AMPL, OPL fundamental improvements in time and space performance possible Ashish Agarwal () 34 / 34

58 Conclusions Automated bigm and convex-hull methods Beginnings of a formalization of probability and statistics Library of transformations Formalization of indexing provides: advances on previous MP languages: e.g. GAMS, AMPL, OPL fundamental improvements in time and space performance possible challenges remain: e.g. conversion of to indexed CNF e i:σ i :σ f :(i:σ σ ) i:σ not supported in current theory. {f (i)/i }e Ashish Agarwal () 34 / 34