Knowledge-Gradient Methods for Efficient Information Collection

Transcription

1 Knowledge-Gradent Methods for Effcent Informaton Collecton Peter Frazer Presentng jont work wth Warren Powell, Savas Dayank, and Dana Negoescu Department of Operatons Research and Fnancal Engneerng Prnceton Unversty Tuesday February 3, 200 Operatons Research and Informaton Engneerng Cornell Unversty 1 / 3

2 Outlne 1 Overvew of Informaton Collecton Applcatons 2 Global Optmzaton of Expensve Contnuous Functons Problem Descrpton Knowledge-Gradent Polcy Applcaton: Smulaton Model Calbraton (Schneder) Applcaton: Drug Dscovery 3 KG Polces for General Offlne Problems Problem Descrpton and KG Polcy Convergence 2 / 3

3 Outlne 1 Overvew of Informaton Collecton Applcatons 2 Global Optmzaton of Expensve Contnuous Functons Problem Descrpton Knowledge-Gradent Polcy Applcaton: Smulaton Model Calbraton (Schneder) Applcaton: Drug Dscovery 3 KG Polces for General Offlne Problems Problem Descrpton and KG Polcy Convergence 3 / 3

4 Informaton Collecton We consder nformaton collecton problems, n whch we must decde how much and of what type of nformaton to collect. We focus our nterest on sequental Bayesan nformaton collecton problems. In makng such decsons we trade the beneft of nformaton (the ablty to make better decsons n the future) aganst ts cost (money, tme, or opportunty cost). We propose the knowledge-gradent (KG) method as a general way to make nformaton collecton decsons. 4 / 3

5 Secton concludes the paper. Applcaton: 2 RESOURCE Smulaton ALLOCATION PROBLEMS Optmzaton There are many resource allocaton problems n the desgn of dscrete event systems. In ths paper we consder the combnatons s usually very large as we wll show the followng example. 2.1 Buffer Allocaton n Supply Chan Management We consder a 10-node network shown n Fgure 1. There Wefollowng would resource lke allocaton to optmzaton chooseproblem: are 10 servers and 10 buffers, whch s an example of a a staffng supply polcy chan, although n a such hosptal a network could tobe mnmze the model for many dfferent real-world systems, such as a patent watng tme, subject to a cost manufacturng constrant. system, a communcaton or a traffc network. There are two classes of customers wth dfferent Hosptal where 0 s dynamcs a fnte dscrete set under and J: 0 a+ partcular R s a arrval staffng dstrbutons, but polcy the same servce cannot requrements. bewe performance functon that s subject to nose. Often J(@ s consder both exponental and non-exponental evaluated an expectaton analytcally, of some random estmate but we of the can estmate dstrbutons (unform) t va n the network. smulaton. Both classes arrve performance, at any of Nodes 0-3, and leave the network after havng To fnd a good staffng polcy to mplement gone through three dfferent n our stages hosptal, of servce. The routng we s not probablstc, but class dependent as shown n Fgure adaptvely choose whch sequence of 1. Fnte polces buffer szes to at all learn nodes are about assumed whch wth s our where s a random vector that represents uncertan factors exactly what makes our optmzaton problem nterestng. smulator. n the systems. The "stochastc" aspect has to do wth the More specfc, we are nterested n dstrbutng optmally Arrval: C1: Unf[2,18] C2: Exp(O.12) Source: Sh,Chen,and Yucesan 1 Fgure 1: A 10-node Network n the Resource Allocaton Problem 36 / 3

6 Applcaton: AIDS Treatment and Preventon We would lke to treat and prevent AIDS n Afrca. We are uncertan about the effectveness of untred preventon methods, but we can learn about them by usng them n practce. To whch preventon methods should we allocate our resources? How should we balance usng tred and true methods wth usng untred methods that may be better? 6 / 3

7 Applcaton: Product Prcng We would lke to dynamcally prce products to maxmze revenue. We learn about product demand from sales and the prces at whch those sales were made. The nformaton collected depends on the prce: If we prce very hgh, we sell nothng and learn only an upper bound on what people are wllng to pay. If we prce very low, we sell to every vaguely nterested party, but learn lttle about how much they are wllng to pay. 7 / 3

8 More Informaton Collecton Applcatons Desgn a sequence of focus groups to effectvely choose features for a new product to be developed. Choose whch tems n a retal store should carry RFID tags. Decde whether to adopt a new technology now, or to wat and gather more nformaton about how well t works. Manage a supply chan when demand dstrbutons are uncertan, and demand lost due to stockout s unobserved. Desgn an adaptve data collecton strategy that wll quckly and effectvely dentfy the source and extent of radaton contamnaton n an emergency. 8 / 3

9 Example: Rankng and Selecton Introducton Assume we have fve choces, wth uncertanty n our belef about how well each one! wll Now perform. assume we Imagne have fve we choces, canwth make uncertanty a sngle our measurement, belef about how after well each one wll perform. Imagne you can make a sngle measurement, whch we have to make a choce about whch one s best. What should we do? after whch you have to make a choce about whch one s best. What would you do? Warren B. Powell 12 / 3

10 Example: Rankng and Selecton Introducton Assume we! Now haveassume fve choces, we have fve wth choces, uncertanty wth uncertanty our n our belef belef about how well each one wll well perform. each one wll Imagne perform. weimagne can make you can amake sngle a sngle measurement, after after whch you have to make a choce about whch one s best. What whch we have to make a choce about whch one s best. should we would you do? do? No mprovement Warren B. Powell 13 / 3

11 Example: Rankng and Selecton Introducton Assume we! Now haveassume fve choces, we have fve wth choces, uncertanty wth uncertanty our n our belef belef about how well each one wll well perform. each one wll Imagne perform. weimagne can make you can amake sngle a sngle measurement, after after whch you have to make a choce about whch one s best. What whch we have to make a choce about whch one s best. should we would you do? do? New soluton The value of learnng s that t may change your decson Warren B. Powell 14 / 3

12 The Knowledge-Gradent Polcy for Rankng and Selecton The knowledge-gradent polcy values each potental measurement x accordng to value of The measurng knowledge x = E[best gradent we can do wth the measurement] (best we can do wthout the measurement).! Basc prncple:» Assume you can make only one measurement, after whch you have to make a We call ths value fnal choce the(the KGmplementaton factor. decson). It then performs the measurement wth the largest KG» What factor. choce would you make now to maxmze the expected value of the mplementaton decson? Change whch produces a change n the decson.! " # " " $ Change n estmate of value of opton due to measurement Warren B. Powell / 3

13 Outlne 1 Overvew of Informaton Collecton Applcatons 2 Global Optmzaton of Expensve Contnuous Functons Problem Descrpton Knowledge-Gradent Polcy Applcaton: Smulaton Model Calbraton (Schneder) Applcaton: Drug Dscovery 3 KG Polces for General Offlne Problems Problem Descrpton and KG Polcy Convergence 11 / 3

14 Global Optmzaton of Expensve Contnuous Functons We have a functon whose global maxmum we would lke to fnd. We can evaluate the functon wth nose va some black-box, but cannot obtan gradents or other nformaton. Evaluatng the functon s expensve, justfyng the use of a sophstcated algorthm to choose evaluaton ponts. 12 / 3

15 Bayesan Pror on the Functon to be Optmzed We begn wth a Gaussan process pror on f. Under ths pror, our pror belef on the values that f takes on any fnte set of ponts x 1,...,x M s multvarate normal. (f (x 1 ),...,f (x M )) N (µ 0,Σ 0 ), where µ 0 and Σ 0 are functons of x 1,...,x M / 3

16 Updatng the Pror For computatonal reasons, we wll restrct ourselves to a fnte collecton 2 2 of ponts x 1,...,x M. The tme n posteror belef on f (x 1 ),...,f (x M ) s N (µ n,σ n ), where µ n and Σ n can be computed recursvely from t=1, x=0 the parameters of the prevous belef µ n 1 and Σ n 1, the locaton of the tme-n measurement, 0 x n, and the measurement s value, ŷ 0. n x t=4, x=1 ft t=2, x=2 ft ft x t=, x= ft x t=4, t=3, x=1 x= t=1, x=0 ft x x t=7, t=6, x=1x= ft t=2, x x t=, 14 / 3 x

17 Measurement as a Stochastc Optmzaton Problem Our goal s to choose measurements to maxmze our ablty to choose a hgh-value alternatve at mplementaton tme. The reward receved s max µ N. The optmal soluton satsfes Bellman s recurson wth a state varable of S n = (n, µ n,σ n ). V N (S N ) = max µ N, V n (S n [ ) = maxe n V n+1 (S n+1 ) x n = x ]. x However, the state space has O(M 2 ) dmensons, whch makes actually solvng Bellman s recurson mpossble and justfes the search for good heurstc polces. 1 / 3

18 Knowledge-Gradent Polcy The KG polcy assgns a value or KG factor ν n x to each potental measurement x. It then performs the measurement wth the largest KG factor. The KG factor s ν n x = E[best we can do wth the measurement] (best we can do wthout the measurement). max pror (µ n ) posteror ( ) x= alternatves () 16 / 3

19 Knowledge-Gradent Polcy The KG factor s ν n x = E[best we can do wth the measurement] (best we can do wthout the measurement). ] = E n [max x n = x max µ n. max pror (µ n ) posteror ( ) x= alternatves () 1 ) 17 / 3

20 Other Approaches Many other dervatve-free nose-tolerant global optmzaton methods exst, e.g., pattern search, e.g., Nelder-Mead stochastc approxmaton, e.g., SPSA [Spall 12]. evolutonary algorthms, smulated annealng, tabu search response surface methods. The KG method s a Bayesan global optmzaton (BGO) method because t places a Bayesan pror dstrbuton on the underlyng but unknown functon. BGO methods requre more computaton to decde where to evaluate next, but often requre fewer evaluatons to fnd global extrema [Huang et al. 2006]. 18 / 3

21 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 1 / 3

22 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 1 / 3

23 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 1 / 3

24 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 1 / 3

25 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 1 / 3

26 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 1 / 3

27 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 1 / 3

28 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 1 / 3

29 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 1 / 3

30 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 1 / 3

31 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 1 / 3

32 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 1 / 3

33 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 1 / 3

34 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 1 / 3

35 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 1 / 3

36 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 1 / 3

37 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 1 / 3

38 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 1 / 3

39 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 1 / 3

40 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 20 / 3

41 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 20 / 3

42 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 20 / 3

43 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 20 / 3

44 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 20 / 3

45 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 20 / 3

46 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 20 / 3

47 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 20 / 3

48 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 20 / 3

49 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 20 / 3

50 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 20 / 3

51 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 20 / 3

52 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 20 / 3

53 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 20 / 3

54 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 20 / 3

55 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 20 / 3

56 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 20 / 3

57 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 20 / 3

58 Computng the KG Factor max pror (µ n ) posteror ( ) x= alternatves () ) best posteror (max observaton ( ) 20 / 3

59 Computng the KG Factor max! n+1 a 1 +b 1 y a 2 +b 2 y a 3 +b 3 y a 4 +b 4 y observaton ( ) The KG factor νx n for measurng alternatve x s ] νx n = E n [max x n = x max x ( ) a+1 a = (b +1 b )f, j b +1 b 2008 Warren B. Powell 1 where f (z) = ϕ(z) + zφ(z), ϕ s the normal pdf and Φ s the normal cdf. µ n x 21 / 3

60 Maxmzng the KG Factor We compute the KG factor for each canddate measurement, and choose the measurement wth the largest. 8 7 µ n µ n +/ sqrt(σ n ) alternatves () 0 log(kg factor) alternatves () 22 / 3

61 Example 23 / 3

62 Example 23 / 3

63 Example 23 / 3

64 Smulaton Model Calbraton at Schneder Natonal The logstcs company Schneder Natonal uses a large smulaton-based optmzaton model to try what f scenaros. The model has several nput parameters that must be tuned to make ts behavor match realty before t can be used. Schneder Natonal 2008 Warren B. Powell Slde Warren B. Powell Slde / 3

65 Smulaton Model Calbraton at Schneder Natonal Current company practce gets company drvers home 2 tmes per month, and ndependent contractors 1.7 tmes per month, on average. The optmzaton model awards a bonus to tself each tme t brngs a truck drver home. Goal: adjust the bonuses to make the optmal soluton found by the model match current practce. Runnng the smulator to convergence for one set of bonuses takes 3 days, and the full calbraton takes 1 2 weeks when done by hand. 2 / 3

66 Smulaton Model Calbraton Results 3 Mean of Posteror, µ n 3 Std. Dev. of Posteror Bonus Bonus Bonus 1 3 log(kg Factor) Bonus 1 0 Best Ft Bonus log10(best Ft) Bonus n 26 / 3

67 Smulaton Model Calbraton Results 3 Mean of Posteror, µ n 3 Std. Dev. of Posteror Bonus Bonus Bonus 1 3 log(kg Factor) Bonus 1 0 Best Ft Bonus log10(best Ft) Bonus n 26 / 3

68 Smulaton Model Calbraton Results 3 Mean of Posteror, µ n 3 Std. Dev. of Posteror Bonus Bonus Bonus 1 3 log(kg Factor) Bonus 1 0 Best Ft Bonus log10(best Ft) Bonus n 26 / 3

69 Smulaton Model Calbraton Results 3 Mean of Posteror, µ n 3 Std. Dev. of Posteror Bonus Bonus Bonus 1 3 log(kg Factor) Bonus 1 0 Best Ft Bonus log10(best Ft) Bonus n 26 / 3

70 Smulaton Model Calbraton Results The KG method calbrates the model n approxmately 3 days, compared to 7 14 days when tuned by hand. The calbraton s automatc, freeng the human calbrator to do other work. Current practce uses the year s calbrated bonuses for each new what f scenaro, but to enforce the constrant on drver at-home tme t would be better to recalbrate the model for each scenaro. Automatc calbraton wth the KG method makes ths feasble. 27 / 3

71 Drug Dscovery We are workng wth a medcal group at Georgetown Unversty hosptal to mprove upon a small molecule they beleve can treat Ewng s sarcoma. As test cases, we use other famles of molecules for whch data has been collected and publshed, ncludng the benzomorphan famly at rght. We use the Free-Wlson model, under whch a molecule s value s the sum of the values of ts substtuent-locaton pars. Source: Katz, Osborne, Ionescu 177 Benzomorphan molecule, wth locatons R1-R avalable for substtuton. 28 / 3

72 Drug Dscovery: Numercal Results. 4. Qualty of Best Molecule log(kg/mol) best value KG Explore PE measurements Number of Measurements Best Value KG Polcy Exploraton Polcy 2 / 3

73 Outlne 1 Overvew of Informaton Collecton Applcatons 2 Global Optmzaton of Expensve Contnuous Functons Problem Descrpton Knowledge-Gradent Polcy Applcaton: Smulaton Model Calbraton (Schneder) Applcaton: Drug Dscovery 3 KG Polces for General Offlne Problems Problem Descrpton and KG Polcy Convergence 30 / 3

74 The General Offlne Informaton Collecton Problem 1 We begn wth a pror dstrbuton on some unknown truth θ. 2 We make a sequence of measurements, decdng whch types of measurement to make as we go. 3 After N measurements, we choose an mplementaton decson and earn a reward R(θ,). In the global optmzaton problem prevously dscussed, θ s the functon whose optmum we seek. Ths functon s doman s the same as both the spaces of possble measurement types x and possble mplementaton decsons. R(θ,) = θ(). 31 / 3

75 The KG Polcy for General Offlne Problems The KG polcy for any problem from ths general framework s ] arg maxe n [max x x n = x max µ n. µ n = E n [R(θ;)] s the expected value of mplementaton decson gven what we know at tme n. s defned smlarly. Evaluatng ths expresson for the KG decson s often computatonally ntensve. 32 / 3

76 Optmalty and Convergence Results The KG polcy s myopcally optmal n general (optmal when N = 1). In certan specal cases (e.g., ndependent normal rankng and selecton on 2 alternatves) the KG polcy s optmal for all N. In many problems, the KG polcy s provably convergent. Convergence means that the alternatve we thnk s best, argmax E N [R(θ,)], converges to the one that s actually the best, argmax R(θ,). In the global optmzaton of expensve functons, convergence means the KG polcy always fnds the global maxmum when gven enough measurements. KG s n some sense a myopc polcy, and so convergence s mportant because t shows that myopa does not mslead KG nto gettng stuck, measurng one alternatve over and over. 33 / 3

77 Concluson Knowledge-gradent polces form a broadly applcable class of nformaton collecton polces wth several appealng propertes: KG polces are myopcally optmal n general. KG polces are convergent n a broad class of problems. KG polces perform well numercally aganst other exstng polces n several problems. KG polces are flexble and may be computed easly n a broad class of problems. 34 / 3

78 Thank You Thank you. Any questons? 3 / 3