Discrete Simultaneous Perturbation Stochastic Approximation on Loss Function with Noisy Measurements

Transcription

1 0 Amercan Control Conference on O'Farrell Street San Francco CA USA June 9 - July 0 0 Dcrete Smultaneou Perturbaton Stochatc Approxmaton on Lo Functon wth Noy Meaurement Q Wang and Jame C Spall Abtract Conder the tochatc optmzaton of a lo functon defned on p-dmenonal grd of pont n Eucldean pace We ntroduce the mddle pont dcrete multaneou perturbaton tochatc approxmaton () algorthm for uch dcrete problem and how that convergence to the mnmum acheved Content wth other tochatc approxmaton method th method formally accommodate noy meaurement of the lo functon Keyword Stochatc optmzaton; recurve etmaton; SPSA; noy data; dcrete optmzaton I INRODUCION HE optmzaton of real-world tochatc ytem typcally nvolve the ue of a mathematcal algorthm that teratvely ee out the oluton It often the cae that the doman of optmzaton dcrete Reource allocaton for ntance nvolve the dtrbuton of dcrete amount of ome reource to a fnte number of uer n the face of uncertanty; other problem of nteret wthn th framewor nclude weapon agnment plant locaton networ reource and expermental degn h paper ntroduce a method for tochatc dcrete optmzaton that baed on tochatc approxmaton technque cutomarly ued n contnuou optmzaton problem Many method have been propoed to deal wth dcrete optmzaton problem hee method nclude random earch [] mulated annealng [] tochatc comparon [7] ordnal optmzaton [] neted partton [8] Recently Hannah and Powell [8] propoe an algorthm for one-tage tochatc combnatoral optmzaton problem baed on evolutonary polcy teraton L et al [3] ntroduce a method baed on random earch n the mot promng area propoed n [] And Slenar [9] conder an exhautve local earch method whch degned explctly for noy lo he am here to preent an alternatve method that can fully ue the nformaton of the tructure of objectve functon (eg gradent ) and potentally nvolve fewer functon meaurement he multaneou perturbaton tochatc approxmaton (SPSA) algorthm [0 ] wa Q Wang wth the Department of Appled Mathematc and Stattc of the John Hopn Unverty Baltmore MD 8 USA (e-mal: qwang9@ jhuedu) Jame C Spall wth he John Hopn Unverty Appled Phyc Laboratory Laurel MD USA and wth the Department of Appled Mathematc and Stattc of the John Hopn Unverty Baltmore MD 8 USA (e-mal: jamepall@ jhuapledu) developed for contnuou optmzaton problem of hgh dmenon and where the lo functon expenve to evaluate SPSA a popular algorthm that create gradenttype nformaton from only two noy functon meaurement n each teraton he ncreae n effcency over the fnte dfference tochatc approxmaton method for example ha been hown to be a factor equal to the dmenon of the problem [0] Spall [0] ha condered the convergence of SPSA for three tme dfferentable functon wherea He et al [9] have analyzed the convergence for nondfferentable but contnuou optmzaton Alo Youefan et al [3] have dcued a local randomzed moothng technque for convex nondfferentable contnuou tochatc optmzaton We want to ue a mlar dea of SPSA for the dcrete cae Becaue the uual noton of a gradent doe not apply n dcrete problem t not obvou that the convergence properte demontrated for SPSA hold for the dcrete cae Hll et al [0] conder a dcrete form of SPSA and develop prelmnary reult aocated wth convergence for a eparable dcrete lo functon under pecal condton However th algorthm can be hown to not converge to the optmal oluton n mple example We ntroduce a dfferent form of dcrete algorthm that apple to a broader range of problem whle potentally retanng the eental effcency advantage of tandard SPSA In partcular we ntroduce a mddle pont dcrete multaneou perturbaton tochatc approxmaton () algorthm that apple n a cla of dcrete problem A n conventonal SPSA the method need only two noy meaurement of the lo functon at each teraton Although a full convergence and convergence rate analy ha not yet been conducted we how condton for almot ure convergence of the algorthm to the true parameter value he paper organzed a the follow In Secton II we motvate the general approach by conderng the cae of one dmenonal and decrbe the bac algorthm for general p In Secton III we how that the algorthm converge to the optmal oluton for ome cla of functon under ome condton In Secton IV we how how th algorthm compare wth the localzed random earch method n two example In Secton V we conclude wth a dcuon //$600 0 AACC 450

2 II PROBLEM FORMULAION A Motvaton: One Dmenon Cae Let u frt conder one dmenonal dcrete functon L: where denote the et of nteger { 0 } We want to fnd the mnmal oluton of the lo functon L Let the noy meaurement of the lo functon be y where y L and ndcate the noe Fg how an example of a dcrete functon n one dmenon wth a lne connectng the neghborng nteger pont he lne L can be regarded a a contnuou extenon of L but L a nondfferentable functon at the nteger pont For a pont \ the gradent where g( ) L L and L ( ) L ( ) L ( ) L ( ) the floor functon the celng functon ( ) the mddle pont between and and a Bernoull random varable tang the value Actually ( ) n and n an odd number o ( ) mut be nteger We can ee that g( ) alo well defned at nteger pont and t a ubgradent (a vector γ a ubgradent of L() at f L()L() γ () for all p ) at (() now the mddle pont between and +) hen the etmated gradent for noy functon y( ) y( ) gˆ( ) Ref [9] ha hown that SPSA method tll converge for nondfferentable functon when the functon are contnuou and convex and the doman are convex and compact et L( ) Fg Example of trctly dcrete convex functon and L a contnuou extenon L B Bac Algorthm of Motvated by the pecal example hown above we wll conder the cae when θ p-dmenonal p 3 We have the general bac algorthm a below for functon y L + where L: p and ε noe he bac algorthm : Step0: Pc an ntal gue ˆθ 0 Step: Generate Δ [ p ] where the are ndependent Bernoull random varable tang the value wth probablty Step: ˆ ˆ p π ( θ ) θ where a p-dmenonal p vector wth all component equal unty and ˆ ˆ ˆ θ p Step3: Evaluate y at π( θˆ ) Δ and π( θˆ ) Δ form the etmate of g ˆ ( ˆ θ) ˆ ˆ ˆ ˆ g ( θ ) y( θ ) y ( ) θ where Δ p Step4: Update the etmate accordng to the recuron θ ˆ ˆ ˆ θ a g ˆ ( θ ) In the theoretcal analy below we mae ue of the followng mean gradent-le expreon centered at π( θ ): g ( π( θ)) E L π( θ) Δ L ( ) π θ Δ Δ θ where Δ p-dmenonal vector that ha the ame defnton a Δ mentoned above and may be a random varable n ome cae If each drecton choen equally then g ( π( θ)) ( ) ( ) p L L π θ Δ π θ Δ Δ where ndcate the ummaton over all poble drecton Note that Bernoull cae; we ue Δ Δ and Δ Δ n the Δ to accommodate future extenon to perturbaton dtrbuton other than Bernoull III CONVERGENCE PROPERIES We now preent an almot ure (a) convergence reult for θ ˆ Frt we ntroduce ome defnton that are ued n 45

3 the proof to follow For any pont θ we denote the et of mddle pont of all unt hypercube contanng θ a If θ le trctly nde one unt hypercube contan one pont But f θ le on the boundary contan multple pont For any pont m n we have m t for p where [t t t p ] and m the th component of m Furthermore let ˆ ˆ ˆ { θ0 θ θ } heorem Aume L a bounded functon on p and t ha unque mnmal pont θ Aume alo () a 0 lm a 0 0a and 0a ; () the component of Δ are ndependently Bernoull dtrbuted; () For all [( ) Δ ] 0 a and E the varance of unformly bounded; (v) up ˆ θ a; and (v) gm ( ) ( θ θ ) 0 for all 0 m and all p \{ } hen Proof By the algorthm we have θˆ ˆ ˆ θ a gˆ ( θ ) ˆ θ θ a ˆ ˆ ˆ θ a y ( θ ) y ( ) θ ˆ ˆ ˆ θ a L ( θ ) L ( ) ε ε θ By condton () () (v) and boundedne of L we have ˆ ˆ lm a L( ( θ ) ) L( ( θ ) ) 0 a () Alo uppoe the varance of are Chebyhev nequalty and () () ( ) hen by ( ) lm P a for ome m lm a 0 m m m mplyng by [4 heorem 4] that lm a { } Δ 0 a (3) hrough () we have the relatonhp that ( ) ( ) θ ˆ ˆ ˆ ˆ θ a L L θ θ ε ε and by the reult of () and (3) we get θ ˆ ˆ θ 0 a Hence there ext uch that θ ˆ ˆ ( ) θ ( ) 0 and P( ) By condton (v) { θ ˆ ( )} a bounded equence for any hen there ext a ubequence { θˆ ( )} and pont θ ( ) uch that { θˆ ( )} θ ( ) In addton we can rewrte () a θˆ θˆ a g( ( θˆ )) ˆ ˆ ˆ a L L g( ( θ )) ( θ ) ( θ ) a ε ε By the defnton of g () we have ˆ ˆ ˆ g ( π( θ )) π( θ ) Δ ( ) π θ Δ Δ E L L ˆ ˆ L ( ) L ( ) p π θ Δ π θ Δ Δ Let ˆ ˆ b L ( ) L ( ) π θ Δ π θ Δ then for all <j we have ˆ ˆ E ( π( θ )) b Δ ( π( θ j )) bjδ j g g ˆ ˆ ( π( θ )) Δ ( π( θ )) Δ E E g b g j bj j j ˆ ˆ ( π( θ )) Δ ( π( θ )) Δ E b E b due to condton () () and (v) for any we have g g j j j j 0 hen ˆ ˆ Δ ˆ Δ g( π( θ )) π( θ ) π( θ ) Δ E a L L a E L L ˆ ˆ Δ ˆ Δ g ( π( θ )) π( θ ) π( θ ) Δ Smlarly due to condton () () and () we have E a ( ) Δ <(4) a E ( ) Δ < (5) Snce for all n ˆ ˆ ˆ a ( π( θ )) L π( θ ) Δ L π( θ ) Δ Δ g and n a ( ) Δ are martngale then by (4) (5) and [3 n heorem 355] we now for all ˆ ˆ ˆ ( ( )) ( ) ( ) a L L g π θ π θ Δ π θ Δ Δ ext and Δ ext Let ( ) a M Δ and a ( ) N ( ( ˆ )) ( ˆ ) ( ˆ ) a L L g π θ π θ Δ π θ Δ Δ then M and N are revere martngale ([ p47]) and by [ heorem 358] there ext random varable M and N uch that M M a and N N a Furthermore n 45

4 due to (4) and (5) we have lm E N lm E M 0 Alo EM 0 and lm 0 and lm EN 0 hen M0 a whch ndcate M 0 a and N0 a whch ndcate N 0 a hen there ext and 3 uch that P( ) P( 3 ) and uch that for any a ( ( ) ( )) Δ 0 and for any 3 ( ( ˆ (ω))) ( ˆ (ω)) ( ˆ (ω)) a L L g π θ π θ Δ π θ Δ Δ 0 Let 4 3 wth P( 4 ) hen for any 4 we have θ(ω) θˆ (ω) a g( π( θˆ (ω))) ˆ ˆ ˆ a L L g( π( θ (ω))) π( θ (ω)) Δ π( θ (ω)) Δ Δ Δ a ε (ω) ε (ω) mplyng ε (ω) ε (ω) a ˆ ˆ ˆ a g L L 0 and π θ π θ Δ π θ Δ Δ 0 a In addton we now{ θ ˆ ( )} θ ( ) ( ( (ω))) ( (ω)) ( (ω)) ndcatng that a g ( π( θˆ (ω))) 0 a (6) Becaue θˆ (ω) θ (ω) then for any>0 there ext S>0 uch that when >S θˆ (ω) θ(ω) < hu there exts when >S all π( θˆ (ω)) We now how θ(ω) the optmal pont By way of contradcton uppoe θ (ω) not the optmal oluton hen by condton (v) we have gm ( ) ( θ( ) θ ) 0 for all m whch a contradcton of g ( π( θˆ (ω))) 0 when >S hen a for all 4 the lmtng pont of the equence { θ ˆ ( )} unque whch equal to θ hu ˆ θ θ a Comment : he nner product condton (v) a natural extenon of the tandard nner product condton for contnuou problem (eg [ p06]) whch nclude convex functon a a pecal cae Comment : Actually ome people have condered the dcrete convexty Mller [4] a forerunner n the early 970 n the area of dcrete convex functon Ref [4] ha ntroduced the defnton of dcrete convex functon and howed that the local optmal pont for dcrete convex functon are alo global optmal oluton here are other defnton of dcrete convex functon [5][5][6][6] but [7] how that Mller dcrete convexty contan the other clae of dcrete convexty Note that Mller defnton doe not nclude all functon atfyng condton (v) and condton (v) doe not nclude all functon atfy Mller defnton of dcrete convexty However for p dcrete convex functon atfyng Mller defnton alo atfy (v) he corollare below gve two common functon atfyng condton (v) Even though we decrbe the functon n contnuou form for we only ue ther value at multvarate nteger pont Strctly convex eparable functon mentoned n corollary are dcued n [0] Corollary Strctly convex eparable functon wth mnmal value at multvarate nteger pont atfy the condton (v) n heorem Proof A eparable functon can be wrtten a p L( θ ) L ( ) t where θ [ t t p] And L a dcrete functon ha ame value wth L at multvarate nteger pont Suppoe the unque mnmal pont of L and a multvarate nteger pont wth t t p θ [ ] hen alo the optmal pont of L Becaue t trctly convex then L ( t ) we have for all p \{ } and any ubgradent L ( t )( t t ) 0 for p Moreover for any m gm ( ) p L L m Δ m Δ Δ p L ( m ) L ( m ) p Δ Δ p L ( m ) L ( m ) p Δ Δ p L ( m ) L ( m ) e hen we have p gm ( ) ( θ θ ) L ( m ) L ( m ) ( t t ) Becaue the mnmal pont a multvarate nteger pont then L ( m ) L ( m ) ha the ame gn wth one of the ubgradent of L at t ndcatng that L ( m ) L ( m ) ( t t ) 0 for all p 453

5 hu gm ( ) ( θ θ ) 0 for all m and all p \{ } QED Corollary L a trctly convex pecewe lnear functon wth mnmal value at a multvarate nteger pont and t lnear n each unt hypercube then L atfy the condton (v) n heorem Proof L a dcrete functon that ha ame value wth L at multvarate nteger pont Snce L trctly convex functon then for all p \{ } and for any ubgradent L( θ ) we have L( θ) ( θ θ ) 0 Furthermore for any m g( m) p L L m Δ m Δ Δ p L L m Δ m Δ Δ ( ) p L m Δ Δ where the notaton of gradent hu gm ( ) ( θ θ) L( ) ( ) p m Δ Δ θ θ L( m) ( ) p ΔΔ θ θ L( m ) ( ) θ θ In addton for any m there wll be one ubgradent L( θ ) at pont θ uch that L( m ) L( θ) hen gm ( ) ( θ θ) L( θ) ( θ θ ) < 0 whch ndcate gm ( ) ( θ θ ) 0 for all m and all p \{ } QED IV COMPARISION WIH LOCALIZED RANDOM SEARCH MEHOD Let u now compare the performance of and the localzed random earch method for two lo functon he frt functon condered here a eparable functon p t he econd one a ewed quartc lo functon whch mentoned n [ Ex 66]: L( θ ) B B p 3 p 4 0 ( B ) 00 ( B ) θ θ θ θ where pb an upper trangular matrx of Even though the ewed quartc lo functon doe not atfy condton (v) we wll ee that tll wor for th lo functon We conder the hgh-dmenonal cae for both functon where p 00 and the meaurement noe ε d N(0) Snce the localzed random earch method more effcent n noe-free cae than n noy cae then we wll conder both the noe-free tuaton and noy tuaton he localzed random earch method decrbed n [ Secton 3] whch conder both noe-free lo functon and noy lo meaurement where a threhold parameter τ nvolved We wll retrct the random earch to the cloet neghbor pont and all thee pont are choen wth equal probablty Here for let a a ( A) a 006 (for eparable); a 00 (for ewed quartc) A For the localzed random earch method we chooe for the noy cae after everal tunng he ntal gue et to be 000 n all run Fg and 3 how the performance of both method under noe-free and noy tuaton for eparable functon And Fg 4 and 5 how the performance of both method for a ewed quartc functon We can ee that doe better than the random earch method for thee two example ˆ θ θ ˆ θ0 θ Number of Meaurement Fg Performance of localzed random earch method and under noe-free tuaton for eparable functon ˆ θ θ ˆ θ0 θ Number of Meaurement Fg 3 Performance of localzed random earch method and wth noy meaurement for eparable functon 3 09 ˆ 08 θ θ 07 ˆ θ0 θ Number of Meaurement Fg 4 Performance of localzed random earch method and under noe-free tuaton for ewed quartc functon 454

6 ˆ θ θ ˆ θ0 θ Fg 5 Performance of localzed random earch method and wth noy meaurement for ewed quartc functon V CONCLUSION Number of Meaurement In th paper we ntroduced a dcrete SPSA algorthm and preented ome prelmnary convergence analy A prelmnary numercal tudy how that wor well on hgh-dmenonal problem wth or wthout noe n the lo meaurement A part of future wor we plan to formally tudy the convergence rate of the and conder non-bernoull random varable for the perturbaton vector Alo we ntend to compare wth other popular dcrete optmzaton algorthm ncludng thoe degned explctly for handlng noy lo meaurement (eg [7][3][9]) wo mportant practcal problem of nteret that nvolve tochatc dcrete optmzaton are reource allocaton where a fnte amount of a valuable commodty mut be optmally allocated and expermental degn where t neceary to chooe the bet ubet of nput combnaton from a large number of poble nput combnaton n a full-factoral degn (eg [4]) We ntend to explore the applcaton of to thee or other problem ACKNOWLEDGMEN h wor wa upported n part by the JHU/APL IRAD Program REFERENCES [] M H Alrefae S Andradóttr A Smulated Annealng Algorthm wth Contant emperature for Dcrete Stochatc Optmzaton Management Sc vol 45 No5 May 999 pp [] S Andradóttr A Method for Dcrete Stochatc Optmzaton Management Sc vol 4 No December 995 pp [3] P Bllngley Probablty and Meaure Wley-Intercence hrd Edton 995 [4] K L Chung A Coure n Probablty heory Academc Pre hrd Edton 00 [5] P Favat F ardella Convexty n Nonlnear Integer Programmng Rcerca Operatva vol pp 3 44 [6] S Fujhge K Murota Note on L-/M-convex Functon and the Separaton heorem Mathematcal Programmng vol pp 9 46 [7] W B Gong Y C Ho W Zha Stochatc Comparon Algorthm for Dcrete Optmzaton wth Etmaton SIAM J Optm vol 0 No 000 pp [8] L A Hannah and WB Powell Evolutonary Polcy Iteraton Under a Samplng Regme for Stochatc Combnatoral Optmzaton IEEE ranacton on Automatc Control vol 55 No5 May 00 pp [9] Y He M C Fu and S I Marcu Convergence of Smultaneou Perturbaton Stochatc Approxmaton for Nondfferentable Optmzaton IEEE ranacton on Automatc Control vol 48 No8 Augut 003 pp [0] S D Hll L Gerencér and Z Vágó Stochatc Approxmaton on Dcrete Set Ung Smultaneou Dfference Approxmaton Proceedng of the 004 Amercan Control Conference Boton MA June 30July 004 pp [] Y C Ho Q C Zhao and Q S Ja Ordnal Optmzaton: Soft Optmzaton for Hard Problem Sprnger New Yor NY 007 [] L J Hong and B L Nelon Dcrete Optmzaton va Smulaton Ung COMPASS Oper Re vol 54 No 006 pp 5 9 [3] J L A Sava and X Xe Smulaton-Baed Dcrete Optmzaton of Stochatc Dcrete Event Sytem Subject to Non Cloed-Form Contrant IEEE ranacton on Automatc Control vol 54 No December 009 pp [4] B L Mller On Mnmzng Noneparable Functon Defned on the Integer wth an Inventory Applcaton SIAM Journal on Appled Mathematc vol No July 97 pp 6685 [5] K Murota Dcrete Convex Analy Mathematcal Programmng vol pp [6] K Murota A Shoura M-convex Functon on Generalzed Polymatrod Mathematc of Operaton Reearch vol pp [7] K Murota A Shoura Relatonhp of M-/L- Convex Functon wth Dcrete Convex Functon by Mller and Favat-ardella Dcrete Appled Mathematc vol 5 00 pp 5 76 [8] L Sh and S Olafon Neted Partton Method for Global Optmzaton Oper Re vol 48 No3 000 pp [9] J Slenar P Popela Integer Smulaton Baed Optmzaton by Local Search Proceda Computer Scence vol 00 pp [0] J C Spall Multvarate Stochatc Approxmaton Ung a Smultaneou Perturbaton Gradent Approxmaton IEEE ranacton on Automatc Control vol 37 No3 March 99 pp [] J C Spall An Overvew of the Smultaneou Perturbaton Method for Effcent Optmzaton John Hopn APL echncal Dget vol 9 No4 998 pp [] J C Spall Introducton to Stochatc Search and Optmzaton: Etmaton Smulaton and Control Wley Hoboen NJ 003 [3] F Youefan A Nedć and U V Shanbhag Convex Nondfferentable Stochatc Optmzaton: A Local Randomzed Smootng echnque Proceedng of the Amercan Control Conference Baltmore MD June 30 July 00 pp [4] J C Spall Factoral Degn for Choong Input Value n Expermentaton: Generatng Informatve Data for Sytem Identfcaton IEEE Control Sytem Magazne vol 30 no 5 October 00 pp