Slice Sampling Approaches for Stochastic Optimization

Transcription

1 Slice Sampling Approaches for Stochastic Optimization John Birge, Nick Polson Chicago Booth INFORMS Philadelphia, October 2015 John Birge, Nick Polson (Chicago Booth) Split Sampling 05/01 1 / 28

2 Overview Epectations, Normalization, and Optimization Epectations, Normalization, and Optimization Nested Sampling, Serial tempering, Bridge Sampling, Wang-Landau, Cross-Entropy, Conditional Probability Estimator,..., Stochastic Optimization Models Single-stage (decision and outcome) Two-stage (decision, outcome observation, recourse action) Multi-stage (repeated decision-observation sequences) John Birge, Nick Polson (Chicago Booth) Split Sampling 05/01 2 / 28

3 Broad view: Importance Sampling Importance sampling is a variance reduction technique. Given an importance blanket g(): E (L()) 1 N Posterior g() = L()π()/Z. N L( (i) ) π((i) ) n=1 g( (i) ) where (i) g() Reduce the problem to finding a posterior normalization constant π L () = L()π()/Z where Z = L()π()d X Harmonic Mean: Ẑ 1 = 1 N N n=1 L 1 ( (i) ). John Birge, Nick Polson (Chicago Booth) Split Sampling 05/01 3 / 28

4 One-stage Problems The set-up for the 1-stage problem is as follows: Find such that E ω [G (ω, )] = ma E ω [G (ω, )], (1) where ω p(ω) for G (, ) given. Let G () = E ω [G (ω, )]. Assuming G > 0 and E ω [G (ω, )] dµ() < for some measure µ(d) to avoid singularities for. Let π J (ω J, ) J j=1 G (ω j, )p(ω j ). The marginal π J () E ω [G (ω, )] J has its mode at and collapses on as J as required (see Pincus (1968)). John Birge, Nick Polson (Chicago Booth) Split Sampling 05/01 4 / 28

5 Finding Conditionals The MCMC conditionals can then be found as: and π J (ω j ) G (ω j, )p(ω j ) π J ( ω J ) J j=1 G (ω j, )p(ω j ), which can be sampled through Gibbs sampling or via Metropolis-Hastings. The result is then the Markov chain with samples, {ω J,(h), (h) } H h=1. Key properties: Can simulate the ω j s from a density that depends on current state in the chain (h) Convergence: is achieved with {ω J,(H), (H) } π J (ω J, ) and (H) in mode (and epectation as J ) No optimization steps or step-lengths No dependence on functional properties (e.g., conveity) John Birge, Nick Polson (Chicago Booth) Split Sampling 05/01 5 / 28

6 Eample: Portfolio Optimization Setup: G (ω, ) = K e γ(r(ω)t +r f ), where r(ω) is an n-vector of the ecess returns of the risky assets relative to a riskfree return r f and K is chosen sufficiently large that the returns can be restricted so that G (ω, ) 0. Analytical solution: For r normally distributed with mean µ and covariance Σ, = Σ 1 µ/γ. MCMC implementation: Introduce an auiliary variable w so that G (ω, ) = r(ω) T +r f log K /γ γe γw dw, i.e., so that w is conditionally a truncated eponential random variable. We then have π J (ω J, ) J j=1 γe γw j 1 { log K /γ wj r T j +r f } e (r µ)t Σ 1 (r µ)/2. John Birge, Nick Polson (Chicago Booth) Split Sampling 05/01 6 / 28

7 MCMC Iterations for Portfolio Eample The MCMC iterations then iteratively draw w j, returns r j, and solutions (g) as follows. w j log(k )/γ E(γ, [0, log(k ) γ + rj T + r f ])(truncated eponential), r j N ((µ, Σ) rj T + r f w j )(truncated normal), w j r T j k k U([ma k r f w j r T j k, min k r f ])(uniform), r jk >0 r jk r jk <0 r jk where k = 1,..., n and subscript k denotes the vector of components other than k. John Birge, Nick Polson (Chicago Booth) Split Sampling 05/01 7 / 28

8 Portfolio Results ( ) Histogram of 1 values for n = 2, µ = 0.05, r f = 0, Σ =, K = 2, and J = 100 for the last 1000 iterations in a 5000 iteration sequence. Note: = (0.833, 0.833) T. The modal intervals for 1 and 2 both include the corresponding value. Histogram of ans$[ (1:4000), 1] Frequency John Birge, Nick Polson (Chicago Booth) Split Sampling 05/01 8 / 28 1

9 Two-Stage Stochastic Programming Problem Setup: Find to solve where S = {A = b, 0} and Q(, ω) = min S ct + E ω [Q(, ω)] min y(ω) 0 {qt (ω)y Wy = h(ω) T (ω)}, y(ω) is known as the recourse decision given the realization of ω. Conversion to one-stage problem: Note that by duality, Q(, ω) = G (h(ω) T (ω)) where G (h T) = ma{(h T) T ξ : W T ξ q}. ξ Two-stage problem is then equivalent to: [ ma S E ω ma {(h(ω) T (ω)) T ξ + c T } ξ:w T ξ q(ω) ] John Birge, Nick Polson (Chicago Booth) Split Sampling 05/01 9 / 28

10 MCMC Setup for Two-Stage MOdel Consider the LP problem: G (z) = ma π { π z : W π q } Following Pincus, we consider the annealed distribution p κ (π z, q, W ) = ep ( κπz) I ( W π q ) /Z κ where Z κ = W π q ep ( κπz) dπ is an appropriate normalization constant. As κ, this distribution tends to a Dirac measure on the solution π of the LP. Simulation from p κ (π), however, requires a method for dealing with truncated multivariate eponential distributions. Simulate a Markov chain and obtain draws π (h) for h = 1,..., H and estimate the value function as a Monte Carlo Rao-Blackwellised average G ˆ(z) = 1 H H i=1 π(g) z, an average of piecewise linear functions. John Birge, Nick Polson (Chicago Booth) Split Sampling 05/01 10 / 28

11 Simulating Truncated Multivariate Eponentials The easiest way to draw from a truncated multivariate eponential is to use one-at-a-time Gibbs sampling. This can be slow to converge when there is an icicle in the constraint region. First write = ( +, ) and then transform the system so that all coordinates are positive > 0. Also assume that the constraint is of the form A b where is K 1 and A is n K where typically n K. The constraint A b can be written as a series of conditional constraints (Gelfand, Smith and Lee, 1993) where n a ij j b i implies a ik k b i a ij j ; j=1 j =k therefore, k min i (b i j =k a ij j )/a ik if the a ik > 0. John Birge, Nick Polson (Chicago Booth) Split Sampling 05/01 11 / 28

12 Factorization In some cases, we can use the factorisation p( 1, 2 ) = p( 1 2 )p( 2 ). For eample, consider the joint distribution ep ( q ) I (a a 2 2 b) We can use the transformation z = a 1 1 and y z = a 2 2 ; then, the distribution p(y z) TEp (z,b) (q 2 a 1 2 ). The marginal distribution p(z) is then determined by p(z) e z ( 1 e q 2a 1 2 (b z)) where 0 < z < b. This is a density tilted by a cdf, which can be sampled using the following result. Theorem (Vaduva) Generate X f and Y with cdf F until X Y. This will gives samples from p() = cf ()F (), where c gives the efficiency of the algorithm. John Birge, Nick Polson (Chicago Booth) Split Sampling 05/01 12 / 28

13 Conditioning Multivariate Eponential to the Simple The truncated multivariate eponential on the simple S k 1 is defined by ( ) k 1 f () = d(λ) 1 ep λ j j = ( 1,..., k 1 ) k 1 ; j 0, j 1 j=1 j=1 Kent et al (2004) calculate the normalisation constant for unequal and equal λ s. They write k 1 p K (λ) = d(λ) λ j j=1 ( ) where p K (λ) = P k 1 i=1 X i < 1 is the probability of the indept eponentials lying in the simple. If the λ s are all equal we get ( ) p K (λ) = e λ k 2 λ e λ. j=1 j! John Birge, Nick Polson (Chicago Booth) Split Sampling 05/01 13 / 28

14 Equal λ Setup If the λ j = λ then this distribution can be easily sampled. Consider the transformation j = yr j where y = k 1 j=1 j is the size of. After a change of variables, f (y, r) y k 2 ep ( λy), r S k 2, 0 y 1. We need to sample from the simple and from a truncated gamma as follows. It is straightforward to sample from the simple. Take uniforms U 1,..., U k 2 on [0, 1] and let s 1,..., s k 2 be the successive gaps in the order statistics, s 1 = U 1, s j = U (j) U (j 1) for j = 2,..., k 1. Sampling from a truncated gamma distribution can be done with the ratios of uniforms method.simulating Y Γ(k 1, λ) on [0, 1] is equivalent to X = λy Γ(k 1, λ) on [0, λ]. Then if we let W X min(k 1, λ) be a location shift. Let f (w) be the pdf. Then we can define u +, v, v + and generate (U, V ) uniformly in the rectangle D = [0, u + ] [v, v + ] and accept W = V /U if U < f 1 2 (V /U). John Birge, Nick Polson (Chicago Booth) Split Sampling 05/01 14 / 28

15 Distribution of Y = k 1 i=1 X j The Laplace transform of Y = k 1 i=1 X j where (X j λ j ) Ep(λ j ) is given by M(φ) = E (ep(φy )) = k 1 j= φ/λ j If the λ j are all unequal this can be given by a partial fraction epansion Let a j = k 1 i=1,i =j λ i λ i λ j M(φ) = give the cdf of Y = k 1 i=1 j as k 1 j=1 λ j λ j + φ k 1 i=1,i =j λ i λ i λ j. where the λ s have been ordered. This can be inverted to F (y) = k 1 a j (1 e λ j y ), j=1 Setting y = 1 given the probability of lying in the simple, p T (λ) = k 1 (1 e λ k 1 j λ ) i. j=1 λ i=1,i =j i λ j John Birge, Nick Polson (Chicago Booth) Split Sampling 05/01 15 / 28

16 Eponential Slice Suppose that we wish to find ma X f (, y). Then, we can define the annealed distribution π κ (, y) = ep (κf (, y)) /Z κ where (, y) X If we introduce an auilary eponential slice variable, so that the joint distribution is π(u,, y) κ ep(κu)i ( < u < f (, y)) I ((, y) X ) Then, we have a truncated eponential for the slice variable π(u, y) = κ ep(κu) I ( < u < f (, y)) ep(κf (, y)) and a uniform for (, y) truncated via the bound u < f (, y). Marginalising out u also gives us the appropriate marginal π κ (, y) = ep (κf (, y)) I ((, y) X ) /Z κ assuming this is a well-defined density. John Birge, Nick Polson (Chicago Booth) Split Sampling 05/01 16 / 28

17 MCMC for Two-stage SP For the 2-stage stochastic program, the LP recourse problem can be replaced by an epectation under the annealed distribution p κ (π ω,, q, W ) = ep ( κπ(t (ω) h(ω))) I ( W π q ) /Z κ (ω.) where Z κ is an appropriate normalization constant. We can then sample from this multivariate distribution in a number of ways. For an over-determined W using one-at-a-time Gibbs leads to min/ma constraints. John Birge, Nick Polson (Chicago Booth) Split Sampling 05/01 17 / 28

18 MCMC Sampling for Two-Stage Model Overall, the MCMC method leads to the limit E π ω,,q,w ( π (T (ω) h(ω)) ) (π ) (T (ω) h(ω)) as κ Therefore, we can instead use MCMC methods to solve ) ma E ω ( ma π (T (ω) h(ω)) π:w π q ( ( = ma E ω E π ω,,q,w π (T (ω) h(ω)) )) John Birge, Nick Polson (Chicago Booth) Split Sampling 05/01 18 / 28

19 MCMC Sampling Continued Define a joint distribution p(, π, ω) π q p κ (π ω, )p(ω) where p κ is the annealed distribution. } This has the appropriate marginal p() E ω {E π ω, (π q). We can also introduce a further slice variable to deal with the objective function. Let that slice variable satisfy 0 < u < π q. Consider the augmented joint distribution p(, u, π, ω) I ( 0 < u < π q ) p κ (π ω, )p(ω) with a uniform measure on. The conditional for the π generation essentially adds another constraint I ( 0 < u < π q ) and I ( W π q ). We use one-at-a-time Gibbs to draw from the annealed conditional p(π ω, ) as a truncated multivariate eponential on this set of inequality constraints. The uncertainty variable samples from an eponentially tilted distribution rather than the prior p(ω) in typical stochastic methods; the conditional posterior is p(ω π, ) p(ω) ep ( π (T (ω) h(ω)) ) John Birge, Nick Polson (Chicago Booth) Split Sampling 05/01 19 / 28

20 Conditional Sampling Gelfand, Smith and Lee (1992) propose using Gibbs sampling to do this. Hence you only need the 1-dimensional conditionals p φ (ξ j ξ ( j), a, W, γ), where ξ ( j) denotes the other variables leaving ξ j out. If the inequality constraints W T ξ γ can be written ξ j b ( j) (W, ξ ( j), γ) then the 1-dimensional conditionals are simply truncated eponentials, ξ j ξ ( j), a, W, γ 1 φa j e φa j (ξ j b ( j) ) on ξj b ( j) (W, ξ ( j), γ) In the 2-stage case: a = a(ω, ) and γ = γ(ω, ) and you need to deal with p φ (ξ ω, ) e φa (ω,)ξ 1 {W T ξ γ(ω,)}. now making eplicit the conditioning on (ω, ). The optimum is now a decision function ξ (ω, ). This can be determined by simulating from the joint p φ (ξ, ω, ) = eφa (ω,)ξ C (ω, ) 1 {W T ξ γ(ω,)} John Birge, Nick Polson (Chicago Booth) Split Sampling 05/01 20 / 28

21 Completing the Iteration Here C (ω, ) = W T ξ γ(ω,) eφa (ω,)ξ and is typically not available in closed from. Notice that MCMC uses the following conditionals p φ (ξ j ξ ( j), ω, ) and p φ (ω, ξ) The former does not require the normalisation constant and is just a truncated eponential with parameters depending on (ω, ). The latter p φ (ω, ξ) eφa (ω,)ξ C (ω, ) χ W T ξ γ(ω,) µ(ω, ), however, depends on C (ω, ). We can still avoid the direct evaluation of this by using the Metropolis algorithm and notice that by Clifford-Hammersley we can compute the ratio for any two candidate draws (ω, ) (g) and (ω, ) (g+1) as p φ (ω (g), (g) ξ) p φ (ξ j ξ ( j), ω (g), (g) ) k p φ (ω (g+1), (g+1) ξ) = j=1 p φ (ξ j ξ ( j), ω (g+1), (g+1) ) in terms of the normalization constants of the 1-dimensional conditionals which are known in closed-form. John Birge, Nick Polson (Chicago Booth) Split Sampling 05/01 21 / 28

22 Eample: Farmer Problem from BL 2011 We consider a model based on the crop allocation problem in Birge and Louveau (2011) for the farmer s decision of an initial storage decision and then subsequent allocation decisions y 1 and y 2 that depend on the realization of the random outcome. The overall problem is to maimize profit of k + E [143y y 2 ] subject to storage y 1 + y 2 and investment constraints, 110y y 2 ω 1 and 120y y 2 ω 2, with y 1, y 2 0. This yields conditionals: p(y 1 y 2 ) ep(143κy 1 )I(0 y 1 min((ω 1 30y 2 )/110, (ω 2 210y)/120, y 2 )) p(y 2 y 1 ) ep(60κy 2 )I(0 y 2 min((ω 1 110y 1 )/30, (ω 2 120y 1 )/210, y 1 )) John Birge, Nick Polson (Chicago Booth) Split Sampling 05/01 22 / 28

23 Normalizing Z s We need to calculate the normalizing constant Z (z, ω). The constraint region (y 1, y 2 ) X (ω, ) is given by y 1, y 2 > 0, y 1 + y 2, 110y y 2 ω 1, 120y y 2 ω 2 We will also assume that 0 < < 100, 3000 ω , 10, 000 < ω 2 < Given and (ω 1, ω 2 ), we can find the optimal as we know 0 < y 1 < min ( y 2, (ω 1 30y 2 )/110, (ω 2 210y 2 )/120) For the y variable, we have marginally 0 < y 2 < min(, ω 1 /30, ω 2 /210). Given that < 100 and ω 1 > 3000, this reduces to 0 < y 2 < min(, ω 2 /210). Hence, we can substitute out and consider the nonlinear criteria function E ω ma (60y min ( y 2, (ω 1 30y 2 )/110, (ω 2 210y 2 )/120)) y This leads to a marginal annealed distribution where we use the Pincus result for nonlinear functionals. John Birge, Nick Polson (Chicago Booth) Split Sampling 05/01 23 / 28

24 Solving the 2-stage problem Steps: J = 20 copies of the recourse variables MCMC iterations over, y j 2, and auiliary variables uj For different values of k, with ω 1 U(3000, 500) and ω 2 U(10000, 20000), John Birge, Nick Polson (Chicago Booth) Split Sampling 05/01 24 / 28

25 Solution Values: Effect of κ kappa=0.01, G=10000 kappa=0.1 y y kappa=1 kappa=10 y y John Birge, Nick Polson (Chicago Booth) Split Sampling 05/01 25 / 28

26 Objective Values f k=5 f f f f f k= k=15 k= k=25 k= John Birge, Nick Polson (Chicago Booth) Split Sampling 05/01 26 / 28

27 Histogram of Solutions Histogram of [ (1:niter/2)] kappa=0.001,k=5 Histogram of [ (1:niter/2)] kappa=0.001,k=10 Frequency Frequency Frequency Histogram of [ (1:niter/2)] kappa=0.001,k=20 Frequency Histogram of [ (1:niter/2)] kappa=0.001,k= John Birge, Nick Polson (Chicago Booth) Split Sampling 05/01 27 / 28

28 Conclusions Boltzmann distribution can yield useful form for optimization with MCMC Importance sampling leads to sequential draws of outcomes and actions Adding latent variable and using slice sampling can make draws efficient Two (and multiple) stages can be incorporated Fast convergence in mode to optima (while mean convergence is slower) John Birge, Nick Polson (Chicago Booth) Split Sampling 05/01 28 / 28