SCALED STEEPEST DESCENT METHOD

Transcription

1 SCALED SEEPES DESCEN MEHOD We want to solve the same problem 0. mn x R n f x + x usng another approach, whch we called Scaled Steepest Descent method SSD n short. We propose to solve 0. by tang the safeguarded Barzla Borwen steplength along the scaled steepest descent drecton n each teraton. For smplcty, we focus on a quadratc case 0.2 mn x R n 2 x Hx b x + x where H s an n n postve defnte matrx and b s an n column vector. 0.2 s stll a convex problem snce the superposton of two convex functons s convex. It s well nown that for a convex functon, the local mnmum and the global mnmum concde. We use h x to denote the objectve functon as defned n 0.2. In ths chapter, we frst nvestgate the performance of the proposed SSD method and establsh some convergence results for 0.2. Next, we generalze the SSD method to solvng 0. and compare our method aganst other alternatves n varous settngs.. Motvatons of Our Research Scalng matrx as defned n Chapter 2 s specally desgned to handle optmzaton problems wth L norm nvolved. It s not surprsng that nether Cauchy steplengths nor BB steplengths wor well n the Steepest Descent Drecton snce problem 0.2 s equvalent to a lnear system of nequaltes LOI constraned problem. Drect applcaton of the Cauchy steplength or BB steplength does not guarantee that all the terates are wthn the feasble regon. As a result of the above observaton, both steplengths fal to gve convergence. Inspred by the trust regon method, we propose to use the scaled steepest descent drecton nstead of steepest descent drecton wth BB stepszes. he safeguard mechansm s also ncorporated to avod the chosen stepszes from beng unreasonably too large or small. In secton 2, we propose a varant of the lne search method to solve 0.2. In secton 3, we establsh the framewor of the SSD method. In secton 4, convergence results by smulaton are presented. 2. he lne search based method n the SD drecton We can stll do lne search for the optmal stepsze n the SD drecton but the optmal stepszes are far more complex than the tradtonal Cauchy steplength because of the crossngs of the brea ponts. In the -th teraton, the orgnal Cauchy steplength s gven by and the next terate x + s updated by g α = g g 2 h x g x + = x α g where g = Hx b + sgn x and 2 h x = 2 f x = H.

2 SCALED SEEPES DESCEN MEHOD 2 When we use the above Cauchy steplength α = g g g Hg, the generated sequence x } = sequence does not converge as proved by smulaton. We beleve that the cause of dvergence s that the objectve functon h x s non-dfferentable at each hyperplane x j = 0, j,, n}}. Whenever those hyperplanes are crossed, the correspondng gradent components wll have jumps n values. Hence the smple use of α = g g g Hg result n the ncrease of the objectve functon values because the steepest descent drecton may become an ascent drecton after brea pont crossngs. Nonetheless, we pont out that the correct calculaton of steplength α should follow the steps as gven below: he objectve functon s gven by may h x = 2 x Hx b x + x hence the gradent of h x at x can be represented as g = Hx b + sgn x If we move along the negatve gradent drecton startng from the current terate x, we defne p α h x + = 2 x αg H x αg b x αg + x + = 2 g Hg α 2 g Hx α + g b α + 2 x Hx b x + sgn x αg x αg = h x + 2 g Hg α 2 + g b g Hx α + sgn x αg x αg In order to fnd the mnmum of p α, f we don t have the x + term, we smply get α = g Hx b 2 2 g Hg = g g g Hg snce p α s a quadratc functon of α. Nonetheless we have the extra L norm term here and because of the extra term sgn x αg x αg n p α, to calculate the optmal α s no longer a trval tas. Indeed, smlar to the proof n Chapter 2, p α s pecewse quadratc wth respect to α. he only dfference s that the curvature of h x s always convex. We only have two cases: ether the local mnmum s the mnmum of some quadratc pece after crossng as many non-dfferentable hyperplanes as possble, or the local mnmum s exactly located on one of those non-dfferentable hyperplanes. Note that p α can be rewrtten as p α = h x + = α Hx b d + 2 α2 d Hd + x + αd x + h x hence we get the ncrease ϕ αd for the current terate x as defned by 2. ϕ αd := h x + h x = α Hx b g + 2 α2 g Hg + x αg x Le we had before, β defnes the -th brea pont along d. Suppose that α [ β, β+, 0,, l } where l corresponds to the last brea pont n the drecton d. We have all 3 terms n 2. wth respect

3 SCALED SEEPES DESCEN MEHOD 3 to the brea ponts decomposed as follows: Hx b αd = Hx b [ α β + β β and 2 α2 d Hd = 2 and = Hx b α β d + Hx b [ α β + β β = α β 2 2 d Hd m β m β m = 2 α β 2 d Hd β j βj β j βj β j βj β j βj + + β j βj β j βj + + β β 0 ] + β 0 d d + + β β ] 0 2 d Hd 2 d Hd + α β β j βj d Hd d Hd + + β 2 β β β 0 d Hd 2 d Hd + α β β d Hd + β j βj j=2 β j d Hd x + αd x = x + αd x + βd + x + βd x + β d + x + βd x x = sgn x + β + d + αd x + βd + + sgn x x + βd x = = sgn x + β + d α β d + sgn x + β j + d Defne ϕ j d := g j d + 2 d Hd, j,, l } where g j s the gradent mmedately after crossng the j-th brea pont. hat s, g j := Hx b + sgn x + β j+ d + β j Hd β j βj Hence the ncrease functon ϕ αd s a combnaton of + pecewse quadratc functons ϕ j, j,, + } ϕ αd = ϕ + α β d + ϕ j β j βj d, α [ β, β + Notce that g = f x + sgn x + β + d + β Hd = g + β β Hd 2sgn x j e j n where j corresponds to the axs for the -th brea pont and e j n = [ 0 }} j th component 0 ] We have establshed Lemma to capture the characterstcs of the mnmum wth respect to the brea ponts. d

4 SCALED SEEPES DESCEN MEHOD 4 Lemma. he optmzer α for 2. n the negatve gradent drecton d = g s ether α = β + or α = β 0,, l } + g,g0 d Hd where corresponds to the followng the last brea pont that s crossed, β := max β : β < α } Proof. We examne two cases. Case I: If the optmzer s strctly wthn two consecutve brea ponts,.e., α g 0, g > 0 because of the crossng of the brea pont β, we have ϕ αd α = ϕ + = = g g α β d α + α β d + 2 d + α β µ ϕ j α β β j βj β, + β, ths mples d α 2 d Hd α Substtute d = g 0 nto the above equaton, we can get Bear n mnd that µ = d Hd = g 0 Hg 0 > 0 g α = β +, g0 > β µ therefore the optmal value s ϕ α d = = ϕ j 2 ϕj β j βj β j βj Case II: he mnmum s one of the brea ponts α = β +, 0,, l }. We have g 0, g,, + } snce µ > 0. We further get ϕ α d = ϕ j 2 β j βj β j βj d β + g j, g 0 g d, 2 g0 + g, g0 µ 2 µ 2 µ g 0 g, 2 g0 2 µ β, βl }. Wthout loss of generalty, we assume,, g 0, g > 0 and β j + gj,g 0 µ > β j, j β g, g 0 + β + 2 β + 2 β g, g 0 β β + β µ }} β + β µ g,g0 he ntuton behnd Lemma s qute straghtforward. After we cross some brea pont, the steepest descent drecton defned at x may become an ascent drecton hence the mnmum of 2. may be some brea pont n the drecton g 0. We propose our verson of lne search based algorthm as elaborated below:

5 SCALED SEEPES DESCEN MEHOD 5 Algorthm Lne Search Based Algorthm Gven x 0, for = 0,, Step. Whle termnatng condtons are not satsfed do Step 2. Compute h x = 2 x Hx b x + x and g = Hx b + sgn x defne the quadratc model as ϕ αd := h x + αd h x where d = g. Step 3. Compute an optmal steplength α for ϕ αd and bactracng s performed f h x s non-dfferentable at x +. Step 4. Set x + = x + αd and goto step } 3. he Framewor of non-monotone Scaled Steepest Descent Method Algorthm s a monotone lne search based method n solvng 0.2. However, to determne the optmal steplength s computatonally heavy. If ths algorthm s generalzed to solvng 0., second order Hessan matrx nformaton s also needed. Barzla Borwen method s comparable n practcal effcency to conjugate gradent method n solvng the unconstraned optmzaton problem. However, as stated before, we observed that drect applcaton of BB stepsze fals to solve 0.2 snce ths problem s ndeed a constraned problem. In order to deal wth the L norm n 0.2, we choose the scaled steepest descent drecton. he detaled dervaton of the BB stepsze n our stuaton s gven n Secton Dervaton of the BB stepszes. he terates are updated by x + = x α D g mn x, } f x where D dag v x and v x :=. otherwse Note that we have changed the defnton of the scalng matrx. If one grad componet satsfes f x at the currrent terate x and x, then we do not change the correspondng scalng factor. If f x and x <, as descrbed n secton 3.3, the correspondng scalng factor can force the component to be bndng at zero very fast f the stepsze s close. Smlar to the dervaton n [], we want to solve the optmzaton problem as defned by 3. mn α α D x D g 2 = mn α α D x D g α D x D g = mn α D x, D x α 2 2 D x, D g α + D g, D g where x = x x and g = g g. he mnmum of 3. s trvally derved as 3.2 α BB := α = D x, D x D x, D g

6 SCALED SEEPES DESCEN MEHOD 6 Smlarly, the mnmum of mn α D x αd g 2 s gven by 3.3 α BB2 := α = D g, D x D g, D g If we substtute x = x x and g = H x x + sgn x sgn x nto 3.2 and 3.3, we observe that nether BB stepsze nor BB stepsze 2 are Raylegh quotent due to brea pont crossngs. Both BB stepsze and 2 as calculated above sometmes are negatve. We force the chosen stepsze α to be n a predetermned nterval [α mn, α max ] where 0 < α mn < α max to avod negatve stepszes and mae t bounded away from zero Upper-boundedness of the chosen stepsze. In practce, we can set α max to be a constant. We can set As proved n Chapter 2, we only care about the asymptotc propertes of D g 2. If ths norm goes to zero, the frst order necessary optmalty condton s satsfed and the lmt pont of x } = reaches the local mnmum x. For the convex problem h x as defned n 0.2, x ndeed s the global mnmum. We use the orthogonal transformaton matrx = [ ] v v 2 v n such that t transforms H nto a dagonal matrx λ λ 2 H =... λn where v = λ v. It s obvous that g + = Hx + b + sgn x + = H x α D g b + sgn x α D g = Hx b + sgn x α HD g + e where = g α HD g + e e sgn x α D g sgn x = j Ix,x sgn x j. 0 and I x, x + refers to the correspondng axs ndex set of the brea ponts along the drecton D g startng from x to x +. Notce that I x, x + l as defned n the proof and β > 0,,, l } s the -th soluton, n terms of the ncreasng order of the magntude, to the followng system hence β = x + j = x β D g j = 0 x sgnx j j g = j = g j f x j D g j x j g j f x > As before, f v x j = x j, β g. Otherwse, β can approach zero.n

7 SCALED SEEPES DESCEN MEHOD 7 In concluson, D + g + = D + g α HD g + e D + g I α HD F + D + e where we should utlze the recurrence relaton of D + and D. We observe that For dagonal matrx, many matrx norms concde,we use Frobenus norm to derve the norm nequaltes D + g n v x + g 2 and Hence and herefore we get x + = x α D g x + = x α D g x + α D g D + g D g = D g n v x g 2 n v x + g 2 n v x g 2 θ max,,n} + 2α, by observng that f all postve numbers a, b, c satsfy a b and } 2 x, x + α g c,,, n}, then n a n b max,,n} c } v x + g v x g = x + x f x, f x + x f x, f x + > x + f x >, f x + f x >, f x + > + 2α f x, f x + x f x, f x + > x + f x >, f x + f x >, f x + > max + 2α, = max,,n} θ α x } I fx } + max, x + α g } I fx >} }

8 SCALED SEEPES DESCEN MEHOD 8 From D + g + θ α I α HD D g + D + e and f e 0 no sgn changes eventually and θ α I α HD <, then the scaled gradent converges. We need to fnd out the exact formula for θ wth respect to α = D [H x x + sgn x sgn x ], D x x D [H x x + sgn x sgn x ], D [H x x + sgn x sgn x ] If α as defned above s postve, t should satsfy the θ α I α HD < Otherwse, we choose α to be the one satsfyng argmn θ α I α HD : θ α I α HD < } Recap that [ ] I α HD F = tr I α HD I α HD [ ] = tr I α HD α HD + α 2 HD HD = n 2α tr HD + α 2tr [D H HD ] Applyng transform : H H to HD and D 2H H, respectvely and based on the smlar nvarant property of trace operator, we can derve that tr HD = tr D H n = v x λ and tr [ D H HD ] = tr [ D 2 H H ] = n v x 2 λ2 herefore we get [ n ] [ n ] I α HD F = n 2α v x λ + α 2 v x 2 λ2 and D + g D g I α HD F = [ n v x + g 2 n ] [ n n v x g 2 n 2α v x λ + α 2 θ α I α HD F m α Notce that θ α s pecewse lnear once g and x are gven, and the mnmum for the second pece [ n ] [ n ] n 2α v x λ + α 2 v x 2 λ2 s just always located at α = b 2a = [ n v x λ ] [ n ] v x 2 λ2 ] v x 2 λ2

9 SCALED SEEPES DESCEN MEHOD 9 [ n Notce that by Cauchy-Schwartz nequalty, [ n v x λ ] 2 and the mnmum for the norm s c b2 4a = n [ n v x λ ] 2 [ n ] v x 2 λ2 Hence based on the fact that θ α max max,,n} + 2α, ] [ v x 2 λ ] } } I x fx } + max, x + α g } I fx >} s ether a constant over some nterval or monotoncally ncreasng, we can draw a concluson that the global mnmum for θ α should have the followng propertes: m α 2 s pecewse-cubc, and for each pece, t s always postve. 2 In order to fnd the mnmum of m α, frst eep n mnd that θ α s monotone. 3 We hereby propose to use the steplength α = [ n vx λ ] for the purpose to have a steepest [ n vx 2 λ2 ] descent n terms of the magntude of the gradent. In consderaton of that, when the algorthm gets closer to the mnmum, the necessary condtons are almost satsfed, by almost I mean all f x for suffcently large, then D +g D g and by our choce of α, we try to move along the scaled steepest descent drecton such that the scaled gradent exponentally goes down to zero. 4 It does not matter too much f e = 0 as long as D +g D g I α HD F <, the early sgn D + g D g change terms e are sort of exponentally decayed by Π = I α HD F 5 We may be n the stuaton that the mnmum for the Frobenus norm I α HD F s not smaller than at all. If that s the case, t stll shows some drectons on how large at maxmum the steplength should be. In concluson, we can choose α max = [ n [ n α := mn v x λ ] [ n ], max v x 2 λ2 vx λ ] [ n vx 2 λ2 ] α mn, hence we get } D x x, D x x D x x, D [H x x + sgn x sgn x ] 3.3. Constant stepsze even wors when L norm term s domnant. We observe that by smulaton, the sequence generated by x + = x D h x actually converges to the optmal very fast when the L norm s somehow domnant. he reason s smply gven as follows: We can see that by settng the steplength along the scaled steepest descent drecton to be, x x + = g v x = x x g v x = x

10 SCALED SEEPES DESCEN MEHOD 0 Algorthm 2 Scaled Steepest Descent Method Gven x 0,α 0 set = 0, α mn, α max, 0 < τ < τ 2 <, γ 0, and M Z + Step : If D g = 0 stop Step 2: Calculate f max max f x j 0 j mn, M }}, δ g, D g and α mn α max, max α mn, D x x, D x x D x x, D [H x x + sgn x sgn x ] Step 3: Whle f x αd g > f max + γαδ set α new [τ α, τ 2 α], α α new In our mplementaton, we choose α new = τ +τ 2 2 α Step 4: x + = x α D g and goto step }} and we observe that x + = x sgn x g = x f x }} x when the necessary condton f x s satsfed. herefore t s not surprsng the terates converge to the orgn almost exponentally Our Approach: SSDBB Method. From Secton 3. to 3.3, we have already establshed the fundamental recpes of our proposed SSDBB method. he full verson of the algorthm s lsted below: We provde the followng theorem to establsh the convergence result. heorem 2. Assume that Ω 0 = x : h x h x 0 } s a bounded set. Let h : R n R be contnuously dfferentable n some neghborhood N of Ω 0. Let x } be the sequence generated by the SDDBB algorthm Scaled Steepest Descent Method wth Barzla Borwen Steplength. hen ether D j x j g x j = 0 for some fnte j, or the followng propertes hold: lm D g = 0; no lmt pont of x } s a local maxmum of h; f the number of statonary ponts of h n Ω 0 s fnte, the the sequence x } converges. Proof. BC. Remar: Parameter M s used to control the satsfablty of the condton f x αd g > f max + γαδ n the whle loop. As M ncreases, t s more lely that the tral pont gves enough decrease n the objectve values. If M =, then the above algorthm becomes a monotone algorthm.γ s another control parameter to set the hardness of the same condton. Note that the nner product g, D g s negatve. Hence as γ ncreases, the nequalty f x αd g > f max + γαδ s more dffcult to be satsfed. If at some terate x, t s very unlely the f x αd g > f max +γαδ s volated, we wll eep shrnng the stepsze. We don t mpose lower and upper bound on α new n the whle loop because of the above observatons. Otherwse, we may encounter an nfnte loop f we set α new mn α max, max α mn, D x x, D x x D x x, D [H x x + sgn x sgn x ] }}

11 SCALED SEEPES DESCEN MEHOD 4. Smulaton Results In ths secton, we nvestgate the performance of our proposed SSDBB method through smulatons. here are many components n Algorthm 2. Hereby we randomly generate 50 test cases and assess the mpact of each parameter on the convergence rate. We want to choose a set of system parameters wth approprate values so as to gve the best convergence rate. 4.. he mpact of scalng. We prove through smulatons that our scalng technques help to solve the problem 0.2. If the algorthm drectly taes the steepest descent drecton wth BB stepsze, we can see that t does not wor at all he Impact of the system parameters. We examne n dfferent sectons to see the performance nflueced by the settngs of the system parameters he mpact of M. We lst n able to gve an overvew of the overall performance. We choose γ =, α mn = 0.0, α max = α mn, τ = 0., τ 2 = 0.9, ρ = 0. he objectve functon s gven by mn ρ x R n 2 x Hx b x + x and the condton number of H s 3 and the dmenson of the above problem s 0. he termnatng condton s set to be whether or not the dfference between the two consecutve objectve values s less than the typcal tolerance value eps = e 8 or the maxmum number of teratons 000 s reached. We say an executon s unsuccessful though t termnates less than 000 teratons, when the last stepsze s < 0 4. he reason why we choose 0 4 s because we observe that, for successful executons, the norm of the scaled gradent s < 0 4. he success executon should termnate wth the stepsze no less than α mn. If the last steplength s < 0 4, t can easly mae the change n x + = x α D g to be too small so that the dfference n objectve values s less than eps. herefore the algorthm does not converge to the optmal pont. Comments: he faled cases n less than 000 teratons are caused by very small stepszes due to repeated call to step 3. 2 Almost all successful cases are termnated wth stepsze α mn = 0.0 whleas the norm of the scaled gradent s at As M ncreases from to 5, the success rate s tang the maxmum value around M = he mpact of γ. We tae 3 and 4 as the canddates for M. In ths secton, we evaluate the mpact of γ on the success rate. Comments: Smaller γ tends to mae the f condton easer to be volated, that n turn results n earler termnaton from the whle loop n step 3. In theory, the γ 0,. In practce, we observe that f γ 0.3, 0.6, there s no sgnfcant dfference n terms of the success rate and the number of teratons to converge. In our mplementaton, we choose γ = M = 3 or 4 does not mae great dfference. For slow convergence sequence x } =, the dfferent choces of M and γ can not mprove the convergence rate he mpact of weghtng factor ρ. We chec the valdty of our proposed SSDBB method n dfferent scenaros,.e, the change of domnance from L norm to quadratc term. he smulaton results are put n the able 3. We choose M = 3 and γ = 0.5 Comments:

12 SCALED SEEPES DESCEN MEHOD 2 est # M = M = 2 M = 3 M = 4 M = 5 Fal n 2 Fal n 2 Fal n 4 Fal n 9 3 Fal n 3 Fal n Fal n Fal n Fal n Fal n 3 Fal n Fal n Fal n Fal n 9 Fal n Fal n 2 Fal n Fal n 4 Fal n 62 2 Fal n Fal n Fal n 2 Fal n 4 Fal n Fal n Fal n Fal n Fal n Fal n 3 Fal n Fal n Fal n Fal n 2 Fal n Fal n Fal n 2 Fal n Fal n 2 Fal n Fal n 4 Fal n Fal n Fal n Fal n Fal n Fal n Fal n Fal n 4 3 Fal n Fal n 3 Fal n 3 33 Fal n Fal n Fal n 2 Fal n Fal n 2 Fal n 37 Fal n 3 Fal n Fal n 3 Fal n Fal n Fal n 40 Fal n Fal n Fal n 42 Fal n Fal n Fal n Fal n 2 Fal n Fal n 4 Fal n Fal n 2 Fal n 48 Fal n 2 Fal n Fal n Fal n Fal n Success Rate 0% 40% 82% 84% 78% able. Impact of M

13 SCALED SEEPES DESCEN MEHOD 3 est # M = 3, γ = 0.3 M = 3, γ = 0.6 M = 4, γ = 0.3 M = 4, γ = Grad Dverge at 000 Grad Dverge at 000 Grad Dverge at 000 Grad Dverge at Success Rate 90% 86% 88% 88% able 2. Impact of γ on the success rate

14 SCALED SEEPES DESCEN MEHOD 4 est # ρ = 0. ρ = ρ = Grad component dverges Grad component dverges Grad component dverges Grad component dverges Grad component dverges Grad component dverges Grad component dverges Grad component dverges Grad component dverges Grad component dverges Success Rate 98% 94% 56%/80% able 3. Change the role of domnance

15 SCALED SEEPES DESCEN MEHOD 5 In ρ = 0. case, the only falure test case #0 actually gves the norm of the scaled gradent at e level. he stepszes are always chosen to be the lower bound α mn = 0.0 snce the BB stepsze s always out of the bound. he objectve values are at e level as well for all cases. 2 In ρ = case, there are 2 test cases n whch t taes more than 000 teratons to converge to the mnmum even though the stepsze are almost constant around.038. We can conclude that when the L norm s domnant, our algorthm wors qute well. 3 In ρ = 00 case, L norm s less domnant compared to the quadratc term. For those test cases who termnates at 000-th teraton, the norm of the scaled gradent s at level 0 4. here are qute a few cases, 0 out of 50, whose scaled gradent components do not show the tendency to converge to zero. In those cases, the chosen steplength n Step are almost always at the lower bound α mn = 0.0 and the objectve values are less than 0 3. We should bear n mnd that n order to recover the volatlty surface from the observed maret data, L norm should not be neglgble f we want the recovered volatlty surface to possess the stablty property he mpact of the condton number. When the condton number of the matrx H s ncreased, the optmzaton problem 0.2 s more ll-posed. We assess the performance of our SSDBB method n able 4. We choose ρ = 0, M = 3 and γ = 0.5 Comments: In cond# = 0 case, the converged sequences are wth exactly 9/0 components bndng at 0. For the dverged cases, more than component s not bndng at zero. 2 As condton number ncreases, the success rate decreases. Moreover, due to numercal ssues n Matlab, the scaled gradent and the stepsze sometmes have complex values wth mage part to be zero. References [] Jonathan Barzla and Jonathan M. Borwen, wo-po44nt Step Sze Gradent Method n IMA Journal of Numercal Analyss 988, 8, 4 48

16 SCALED SEEPES DESCEN MEHOD 6 est # cond# = 0 cond# = Grad component dverges 5 M = 3 and γ = Grad component dverges Grad component dverges Grad component dverges 2 Grad component dverges 57 3 Grad component dverges Grad component dverges Grad component dverges Grad component dverges Grad component dverges Grad component dverges Grad component dverges Grad component dverges Grad component dverges Grad component dverges Grad component dverges Grad component dverges Grad component dverges Grad component dverges Grad component dverges Grad component dverges 49 Grad component dverges Success Rate 78% 74% able 4. Impact of the condton number

17 SCALED SEEPES DESCEN MEHOD 7 est # ρ = Success Rate 00% able 5. ρ = 00, M = 3 and γ = 0.5 tol=0 8