Practcal Newton s Method Lecture- n Newton s Method n Pure Newton s method converges radly once t s close to. It may not converge rom the remote startng ont he search drecton to be a descent drecton rue the Hessan s Postve Dente Otherwse t may be ascent, or may be ecessvely long wo Strateges: Newton GC: Solve Lnear System usng GC, termnate neg curvature encountered Moded Newton: Mody Hessan beore or durng the soluton
Ineact Newton Stes Iteratve method to solve lnear system, termnate at some aromate soluton. Resdual r r n Scale deendent Relatve Resdual ermnate teratons : + η η, r η η s the orcng seuence How about? η, heorem 6. Suose that s contnuously derentable n a neghborhood o a mnmzer, and assume that s ostve dente. Consder the teraton +, where + satses r η η,, then, the startng ont s sucently near, the seuence { } converges to lnearly. hat s, or all K sucently large, we have: c, c
heorem.7 Newton Lecture-6 Suose that s twce derentable and that Hessan s Lschtz contnuous. Consder the teraton + where s gven by N hen:. I the startng ont s sucently close to, the seuence converges to.. he rate o convergence s uadratc. he seuence o gradent norms converges uadratcally to zero. + r + r I Hessan s PD L aylor Seres L + r L + η + + + O + O r + O L + r + O r η + + r η
+ η + O + + O η + I s chosen close to, we can eect to decrease by a actor o aromately at every teraton. + lm su η < η + O η < I r o lm su + r η I r O + r lm su + O + O + O + O + c
heorem 6. Suose that the condtons o heorem 6. hold and assume that the terates { } generated by the neact Newton method converges to. hen the rate o convergence s suer-lnear η and uadratc η O. Quadratc η mn. 5, r r η r r O + O + O O + r + O + + lm c
Lne-Search Newton-CG Method. he startng ont or GC teraton s. Negatve curvature test. I the search drecton satses A I, comlete the rst GC, comute the new terate, sto I >, sto the rst GC, return most recent soluton. he Newton ste s dened as the nal CG terate gven ntal ont or end,,...,n Set r mn.5, s encountered Algorthm 6. Algorthm 6. Lne Search Newton - CG Comute a search drecton by alyng the CG method to + startng rom +α, whereα satses Wole. ermnate when, or the negatve curvature s bactracng condtons
Problems I Hessan s nearly sngular, Newton-CG drecton can be long, reurng many uncton evaluatons. he reducton n uncton may be very small. Normalze the Newton s drecton Introduce threshold A Algorthm 6. Algorthm 6. Lne Search Newton wth Modcaton gven ntal ont or end,,...,n Factorze the matr Set s sucently PD; otherwse, E ensure that Solve + ; s sucently PD + E, where E s chosen to +α, whereα satses Wole bactracng condtons
ounded Moded Factorzaton Proerty he matrces n the seuence { } have bounded condton number whenever the seuence o Hessan } s bounded, that s: { cond C, or some C >, Hessan Modcaton Choose modcaton E such that matr PD. + E s sucently -modcaton to be well-condtoned -small, so that second order normaton s reserved -modcaton be comutable at moderate cost
Egenvalue Modcaton,,,, dag Λ n Q Q λ Sectral decomoston,, dag and λ λ λ Α Q I.,, N.9 4.. > + N.. N It s not a descent drecton,, 8 + dag λ 8 λ For small ths ste s nearly arallel to and very long. Although decreases along the drecton, ts etreme length volates the srt o Newton s method, whch reles on the uadratc aromaton o the objectve uncton. Relace all negatve egenvalues by small ostve numbers. 8
Fl the sgns o negatve egenvalues, n our case Set Set the last term zero, so that the search drecton has no comonent along the negatve curvature drectons, adat the choce o to ensure the length o the ste s not ecessve. 8 λ