Online Model Racing based on Extreme Performance

Transcription

1 Online Mdel Racing based n Extreme Perfrmance Tiantian Zhang, Michael Gergipuls, Gergis Anagnstpuls Electrical & Cmputer Engineering University f Central Flrida

2 Overview Racing Algrithm Offline vs Online Mdel Selectin Mtivatin Max-Race Algrithm Framewrk Extreme Value Thery & Generalized Paret Distributin Mdeling Inferring Inequivalence f Endpints Algrithm Prtfli based n Max-Race Experimental Results and Discussins Cnclusins and Future Directin 2

3 Racing Algrithm Mdel Selectin Cnsider an ensemble f mdels and stick with the best Brute Frce Apprach Trade-ff Validate the ensemble n a cmmn prblem set Select the champin mdel Benefit: Identify best mdel(s) Price t be paid: Cmputatinal cst RAs trade ff mdel ptimality vs. cmputatinal effrt by autmatically allcating cmputatinal resurces 3

4 Racing Algrithm A set f initial mdels Sample validatin instances Evaluate all mdels Abandn bad perfrmers N Stp? Yes Return the best perfrmers 4

5 Racing Algrithm Gal Cncentrate mre cmputatinal resurces n prmising mdels. Save n verall effrt Cmputatin Time Brute frce Racing algrithms Search Prcess 5

6 Racing Algrithm Name Year Statistical Test Criterin Of Gdness Applicatin Heffding 1994 Heffiding s inequality Predictin Accuracy Classificatin & Func. Apprximatin BRACE 1994 Bayesian Statistics Crss-validatin Errr Classificatin F-Race 2002 Friedman Test Functin Optimizatin Optimal parameter f Max- Min-Ant-System fr TSP Bernstein Racing 2008 Bernstein s inequality Functin Optimizatin Plicy selectin in CMA-ES S-Race 2013 Sign Test Predictin Accuracy Classificatin 6

7 Offline vs Online Mdel Selectin Offline Mdel Selectin Select mdels based n a training set f prblems Pure explratin prblem Explratin: Gather mre infrmatin Online Mdel Selectin Selectin happens during prblem slving Explratin vs. Explitatin Explitatin: Making the best chice based n current infrmatin 7

8 Online Mdel Selectin Classical techniques Reinfrcement Learning (Markv Decisin Prcess) Multi-armed Bandit Algrithm Typical bjective Maximize the expected cumulative r maximum rewards Our bjective Maximize the maximum reward e.g. algrithm selectin fr slving a given NP-hard cmbinatrial ptimizatin prblem e.g. cmputatinal resurces assignment in Ppulatin-based Algrithm Prtfli 8

9 Max-Race - Algrithm Framewrk Suspend-and-Resume A given prblem instance Candidate Mdels Parallel Executin Evaluate all mdels Abandn ptentially bad perfrmers N Stp? Yes Return the final slutin 9

10 Max-Race Extreme Value Thery Statistics f extreme r rare events Analgus t the Central Limit Therem f sample means The sample maximum cnverges apprximately t a Generalized Extreme Value (GEV) distributin. Y = max i=1,,n X i, where X i is a sequence f i.i.d. r.v.s with a cmmn cdf H lim n Pr Y b n a n y = lim n H n a n y + b n where a n > 0, b n are nrmalizing cefficient. = G(y) G(y) is the cdf f a GEV with ε (shape), μ (lcatin), σ (scale) parameters G y = exp 1 + ε y μ σ 1/ε 10

11 Max-Race Extreme Value Thery In the limit, the tail distributin f X i s tends t a Generalized Paret Distributin (GPD) lim Pr X > x X > u = u u F ε,μ,σ(x) where u is the endpint if dist. H, and F ε,μ,σ x = ε(x μ) σ 1 exp x μ σ 1/ε fr ε 0 fr ε = 0 Three types f GPD, crrespnding t three types f GEV distributin 11

12 Max-Race Extreme Value Thery Tw basic appraches t mdeling the parametric distributin f extreme values Blck Maxima (BM) Sample cllected in blcks The maximum f each blck fllws a GEV Pint Over Threshld (POT) Samples exceeding a high threshld fllws a GPD Efficient utilizatin f data Threshld Stability 12

13 Max-Race GPD Mdeling Online mdel selectin Mdels Evlutinary Cmputatin (EC) Algrithms Prblems Functin Maximizatin Samples Best bjective btained every generatin gbest t = max pp t, gbest t 1 A sequence f r.v. f a discrete time Markv chain State f equilibrium gbest t t=t0 are i.i.d. samples Mdel the upper rder statistics f gbest t btained s far by a GPD (with ε < 0, which has a right endpint μ σ/ε) 13

14 CDF Max-Race GPD Mdeling GPD fit fr gbest t cllected frm a CPSO run n a multi-mdel functin ptimizatin Fitted Generalized Paret CDF Empirical CDF lg 10 (x') x 10-9 q=0.5, s=500 14

15 Max-Race GPD Mdeling Threshld selectin Balancing the bias and the variance f parameter estimatin The q-quantile f the sample threshld stability The q value decreases expnentially with the sample size s q s = q 0 e γs q 0 and γ are empirically determined q 100 = 0.7, q 5000 = 0.1 Parameter estimatin f ε, μ, σ Methd f Mments 15

16 CDF CDF CDF Max-Race GPD Mdeling GPD fit fr gbest t cllected frm a CPSO run n a multi-mdel functin ptimizatin Fitted Generalized Paret CDF Empirical CDF Fitted Generalized Paret CDF Empirical CDF Fitted Generalized Paret CDF Empirical CDF lg 10 (x') x lg 10 (x') x lg 10 (x') x q=0.7, s=100 q=0.5, s=500 q=0.9, s=500 16

17 Max-Race Inferring Equivalence f Endpints Hypthesis test t detect differences between sample maxima N cunterparts Tests f parameter value, parameter equivalence, dmain f attractin Inferring equivalence f mdels perfrmance Given bservatins X i i = 1,, n, Y i (i = 1,, m) frm mdel A and B with X i ~F εa,μ A,σ A x, Y i ~F εb,μ B,σ B y, μ A < 0, μ B < 0 H 0 : μ A σ A ε A = μ B σ B ε B (A and B are equally gd) H a : μ A σ A ε A > μ B σ B ε B (A is better than B) 17

18 Max-Race Inferring Equivalence f Endpints The sample maxima is used t estimate the endpints The pdfs are U = max {X 1, X 2,, X n } V = max {Y 1, Y 2,, Y m } f U u = nf εa,μ A,σ A u Fn 1 εa,μ A,σ A (u) f V v = mf εb,μ B,σ B v Fm 1 εb,μ B,σ B (v) The test statistic is the sample maxima difference = U V with pdf f δ = f U u f V u δ du μ A σ A εa μ A 18

19 Max-Race Inferring Equivalence f Endpints Under H 0, σ A ε A, σ B ε B, the p-value is calculated as π d; A, B = P > d where d is the bserved difference between tw sample maxima π d; A, B < α, reject H 0 Numerical integratin Adaptive Gaussian quadrature methd Parameter estimatin Methd f Mments Maximum Likelihd Estimatin and Maximum-Gdness-f-Fit Estimatin 19

20 Max-Race Prtfli (MRP) Initialize algrithm prtfli P A 1, A 2,, A k k 2 Initialize k sub-ppulatins pp 1, pp 2,, pp k k 2 Evlve the current ppulatin fr N cs generatins Recrd all the gbest t P values fr each pair A i, A j P d P values P values {π(d; A i, A j )} end fr P Pl\{A k }, where k FDR P values, α s until maximal NFEs is reached return the best bjective value fund

21 Experimental Results Candidate algrithms PSO, CPSO, jde, JADE, SaDE, CMA-ES Benchmark prblems Multi-mdel maximizatin prblems Gaussian landscape generatr Baseline appraches with fixed NFEs Max-Race vs. Brute Frce apprach (BFA) MRP vs. RandEA & BestEA MRP vs. ther APs AMALGAM-SO PAP MultiEA 21

22 Experimental Results Cmpare Max-Race with BFA PA predictin accuracy RTR rati f running time t cmpletin (t Max Race /t BFA ) N b -- n. f ptimal mdels returned by BFA N r -- n. f ptimal mdels returned by Max-Race dim10 dim30 PA dim10 dim30 PTR dim10 dim30 N b dim10 dim30 N r 22

23 Prbability Density Experimental Results Cmpare MRP with RandEA randmly select an algrithm BestEA always select the best algrithm RandEA BestEA Max-Prtfli f 5 (x) 23

24 Mean Values f Final Outcme Experimental Results MRP vs BestEA vs RandEA Algrithms MRP BestEA RandEA 24

25 Experimental Results Algrithm Prtfli Appraches Cmparisn AMALGAM-SO A Multi-algrithm Genetically Adaptive Methd fr Single Objective Optimizatin Autmatically adjust n. f ffspring fr each sub-ppulatin Elitism selectin PAP Ppulatin-based algrithm prtfli Sub-ppulatin run in parallel, with fixed prtin Ppulatin migratin MultiEA N infrmatin Exchange Only the ptentially best sub-ppulatin is evlved 25

26 Experimental Results Limitatins f Existing Algrithm Prtfli Appraches Inapprpriate selectin f cnstituent algrithms Hw t synchrnize them? Cmputatin time is wasted n inferir algrithms MRP advantages N need t knw the cnstituent algrithms well, s as t cmbine them effectively Limits waste f cmputatinal effrt n under-perfrming algrithms 26

27 Func. 1 Func. 2 Func. 3 Func. 4 Func. 5 Func. 6 Func. 7 Func. 8 Func. 9 Func. 10 Func. 11 Func. 12 Func. 13 Func. 14 Func. 15 Func. 16 Func. 17 Func. 18 Func. 19 Func. 20 Mean Values f Final Outcme Experimental Results & Discussins MRP vs ther Algrithm Prtfli Appraches Algrithms MRP AMALGAM-SO PAP MultiEA 27

28 Cnclusins Max-Race is the first nline racing algrithm, aiming at ptimizing the extreme perfrmance quality while minimizing cmputatinal effrt. A parametric hypthesis test based n the Generalized Paret Distributin is develped t identify significant difference between mdels extreme perfrmances Max-Race is capable f identifying the best algrithms with high likelihd and by incurring lw cmputatinal cst Cmpared with BFA Cmpared with RandEA, Best EA Cmpared with AMALGAM-SO, PAP, MultiEA 28

29 References [1] O. Marn and A. Mre. Heffding races: Accelerating mdel selectin search fr classificatin and functin apprximatin. Advances in Neural Infrmatin Prcessing Systems, 6:59 66, [2] T. Zhang, M. Gergipuls, and G. C. Anagnstpuls. S-Race: A multi-bjective racing algrithm. In GECCO 13 Prc. f the Genet. and Evl. Cmput. Cnf., pages , [3] S. Cles. An Intrductin t Statistical Mdeling f Extreme Values. Springer, [4] P. de Zea Bermudez and S. Ktz. Parameter estimatin f the generalized paret distributin part I. J. Stat. Plan. Inference, 140: , [5] J. Hüsler, P. Cruz, A. Hall, and C. M. Fnseca. On ptimizatin and extreme value thery. Meth. And Cmput. In Appl. Prbab., 5: , [6] C. Scarrtt and A. MacDnald. A review f extreme value threshld estimatin and uncertainty quantificatin. Statistical Jurnal, 10:33 60,

30 Thank yu! Any Questins? 30

31 Back-up Slides 31

32 CDF Max-Race Extreme Value Thery Three types f GEV distributin Fréchet Gumbel Weibull