Particle Filtering for Data-Driven Simulation and Optimization

Transcription

1 Particle Filtering for Data-Driven Simulation and Optimization John R. Birge The University of Chicago Booth School of Business Includes joint work with Nicholas Polson. JRBirge INFORMS Phoenix, October

2 Themes Particle filters can provide an efficient way to analyze and to simulate complex systems Sequential Monte Carlo methods provide an approach to obtain samples from complicated distributions including optima Combining these approaches leads to new methods that can defeat (in some cases) the curse of dimensionality JRBirge INFORMS Phoenix, October

3 Basic problem Outline Particle filtering 001 Sequential methods for simulation Optimizing by simulating (an old idea in a new light) Additional applications Extensions JRBirge INFORMS Phoenix, October

4 Start of Particle Methods Origin: estimating values (distributions) in a hidden Markov model Example: We begin with inventory x 0 (which may be uncertain) We make observations y 1,,y T based on reported sales and new inputs What is the distribution of actual inventory x t at any time t? JRBirge INFORMS Phoenix, October

5 Special Case: LQG Linear, quadratic Gaussian (LQG) Model Implications: All of the distributions are normal Simple updating formulas from Kalman filter: y t = H x t + v t ; x t =F x t-1 + u t ; v~n(0,q); u~n(o,r) Iterations: w t = y t -Hx t ; S=HP H T +R; K=P HTS -1 ; x t =x t +Kw t P =(I-KH)P ; x t =Fx t-1 ; P =FP F T +Q JRBirge RMS Phoenix, October

6 Inventory over 10 periods: - Actual (*) - Observed (+) Kalman estimate: x t ~ N(x t,p t ) Here: x 10 ~ N(13.8,0.62) Kalman Prediction JRBirge INFORMS Phoenix, October

7 Bayesian Interpretation Suppose a distribution on x 0 : f(x 0 ) Assume some transition: p(x t x t-1 ) Observations: q(y t x t ) Goal: find P(x t y 1,,y t ( y T )) Idea: Use Bayes Rule to find the conditional probability: P(x t y t )=P(x t,y t )/P(y t ) / s q(y s x s )p(x s x s -1)f(x 0 ) JRBirge INFORMS Phoenix, October

8 General Processes If the observation and transition functions, q and p are quite general, then P(x t y t ) may not be available analytically (E.g., no sales if stocked out, maximum limit, bulky demand) Alternatives: Markov chain Monte Carlo (MCMC) Particle methods JRBirge INFORMS Phoenix, October

9 Particle Methods Idea: Keep a limited number (N) of particles in each period Propagate forward to obtain new points Re-sample (re-weight) backwards to get consistent weights Re-balance if weights are low References: Gordon, Salmond, Smith (1993), Doucet (2008) JRBirge INFORMS Phoenix, October

10 Propagate w t1, x 1 t w t2, x 2 t... w tn, x N t Method Structure Re-sample: x 1 w 1 t t+1 x 2 w 2 t t+1 y t x N w N t t+1 JRBirge INFORMS Phoenix, October

11 Sequential Importance Resampling (SIR) Method JRBirge INFORMS Phoenix, October

12 Cum. Results N=50 Compare to Kalman Cumulative JRBirge INFORMS Phoenix, October

13 Particle Frequency N=50 Particle (+) Kalman (*) JRBirge INFORMS Phoenix, October

14 Other Uses of Particles Particle filtering: finding x t from y t =(y 1,,y t ) Particle smoothing: finding x t from y T =(y 1,,y T ) Particle learning: (Polson et al.) Find parameters µ t at the same time to explain the model. Simulation JRBirge INFORMS Phoenix, October

15 Particle Filter in Simulation Interpretation: Usage is a form of sequential importance sampling Use q(x t x t-1 ) to draw Many more samples to draw in area of interest (e.g., tail of distribution) Example: Find the probability of a 50% loss over 5 periods for a portfolio with 1-period mean=5%, st. dev.=20% JRBirge INFORMS Phoenix, October

16 Loss Probabilities Compare crude MC to SIS Use q to match target Reduces std. deviation in half Frequency Probability SIS MC JRBirge INFORMS Phoenix, October

17 Earlier in Optimization With particle methods: Taddy, Gramercy, Polson (2008), Jones, Schonlau, Welsh (1998) Pincus (1968): 2 ideas Can use expectation to optimize Can find the expectation with Markov chain JRBirge INFORMS Phoenix, October

18 Pincus results: Using Simulation for Optimization Model: min F(x) Create a distribution P (x)=exp(- F(x)) Then: lim 1 E P (x) = x * JRBirge INFORMS Phoenix, October

19 Markov Chain Simulation From x t to x t+1 Sample over exp(- F(¹ x t +(1-¹) x t+1 ) ) Choose directions to obtain mixing in the region JRBirge INFORMS Phoenix, October

20 Example Rosenbrock function: JRBirge INFORMS Phoenix, October

21 Test for Rosenbrock T=5000 iterations N=500 replications Re-sample when: Effective Sample Size<250 =2, 10, 100 JRBirge INFORMS Phoenix, October

22 Rosenbrock Graphs = JRBirge INFORMS Phoenix, October

23 Other Examples: Rastigrin JRBirge INFORMS Phoenix, October

24 Himmelblau JRBirge INFORMS Phoenix, October

25 Shubert JRBirge INFORMS Phoenix, October

26 Combine Particles and Simulation Suppose N particles are used Consider a 2-stage problem f(x,») = c(x) + Q(x,y,») Distribution on x, y i,»: P(x,y 1, y N,»)= i=1 N (c(x)+q(x,y i,» i )) Result: E P (x) x * JRBirge INFORMS Phoenix, October

27 Implementation Propagation forward on y s: q(» j x 1 ) / { c(x) + Q(» j, y j, x)} p(» j ) where y j =argmin(c(x) + Q(» j, y j, x)) p( x» 1,y 1,,» Ny N ) / j=1 N {c(x) + Q(» j, y j, x) }p(» j ) The particles concentrate the distribution on x* as in the deterministic optimization. JRBirge INFORMS Phoenix, October

28 Example From B/Louveaux, Chapter 1 (Farmer): JRBirge INFORMS Phoenix, October

29 Example: Linear-Quadratic Gaussian All distributions are available in closed form x t =Ax t-1 +B t u t-1 +v t All normal distributions with risk-sensitive objective: -log E(exp(-(µ/2)(x 2T Qx 2 +u 1T Ru 1 ))) where u 1 is control in the first period Now, all can be found in closed form JRBirge INFORMS Phoenix, October

30 Results for LQG Mean of u: m u 1 m u 1 = L 1(µ) x 1, where L 1 is the result from the Kalman filter Variance of u: V u (N) where (N) 0 as N 1 V u 1 m u 1 (µ) u 1 * as µ 0 where u 1 * is the two-stage risk-neutral solution. JRBirge INFORMS Phoenix, October

31 Extensions to T Stages Can derive S t µ recursively to show that: p(x t x t-1,u t-1 )=K exp(-(1/2) ( s=1 t-1 (x tt Q s x s +u st R s u s )+x tt S t µ x t ) + (x t -Ax t-1 -Bu t-1 ) T V t-1 (x t -Ax t-1 -Bu t-1 )) For particle methods, j=1 N: Each x t j drawn consistently from this distribution JRBirge INFORMS Phoenix, October

32 w t1, x t 1 u t 1 w t2, x t 2. u t 2 w tn, x t N u t N General Method Structure Propogate Re-sample: x 1 t+1 w t1, x 1 t x 2 t+1 u 1 t... x t+1 N w t2, x t 2 u t 2 JRBirge INFORMS Phoenix, October w tn, x t N u t N

33 General Result If the propagation step is unbiased, then E(u 1 ) u 1 * The unbiased result is possible in LQG from the recursive structure. Harder to obtain in more general systems (without additional future sampling) JRBirge INFORMS Phoenix, October

34 Generalizing Objectives Direct: risk-sensitive exponential utility: exp(-µ Q(x,y,»)) - Q quadratic Objective: e -µ y - d E [exp((-µ 2 /2 j)(y j d) 2 )] where ~ Exp(2) Also, use: E[ e -µ y - d ]¼ 1 + µ y-d + O(1) JRBirge INFORMS Phoenix, October

35 Further Generalizations Use objective approximations that preserve normal forms Extend LQG-like results to this general framework Find other frameworks where unbiased future samples are available Obtain sensitivity results on bias from imperfect future sampling Use optimization to pick ML instead JRBirge INFORMS Phoenix, October

36 Conclusions Estimation models have many similar aspects to optimization models (e.g., often MLE) Particle methods allow for general forms of estimation in time series when analytical results are not available Simulation on a larger distribution including decision variables can combine the methods Key feature to remember is the backward step of updating controls with observations JRBirge INFORMS Phoenix, October

37 Other Questions to Consider How can learning and parameter estimation be combined with optimization? What is the range of problems that can be approached in this manner? What are the best uses of optimization technology as part of this process? JRBirge INFORMS Phoenix, October

38 Thank you! Questions? JRBirge INFORMS Phoenix, October