ELE 538B: Large-Scale Optimization for Data Science. Quasi-Newton methods. Yuxin Chen Princeton University, Spring 2018

Transcription

1 ELE 538B: Large-Scale Opimizaion for Daa Science Quasi-Newon mehods Yuxin Chen Princeon Universiy, Spring 208

2 00 op ff(x (x)(k)) f p 2 L µ f 05 k f (xk ) k f (xk ) =) f op ieraions converges in only 5 seps X k=0 ypically requires quadraic local convergence f (x ) x k22 kx f op L2f 2L2f P µ µ( + ) k=0 k x k22 + µ( + ) + kx k=0 () f op 0 Summing over all ieraions before, we ge X quadraic backracking parameers αconvergence: = 0., β = 0.7 and hence ) µ( + ) kx+ x k22 + µ( + ) + µ( x R ) n f (x) Examplesminimize f (x ) f op kx x k22 kx + 2 x = x ( f (x )) f (x ) Summing over all ieraions before, we ge f bes,k k=0 X k 2 kg k2 µ 2 kg k2 µ x k22 + µ( + ) + kx x k22 µ( ) kx x f op kg k22 µ( + ) and hence xample in R2 (page 0 9) x(0) µ( Newon s mehod f (x ) =) µ( + ) + kx kg k2 µ x k22 + L2f µ f bes,k x k22 + X k 2 kg k2 µ k=0 L2f 2L2f P µ µ( + ) k=0 k k f (x ) 5 f op aains ε accuracy wihin O(log log ε ) soring and invering Hessian 2 f (x) Rn n a single ieraion may las forever; prohibiive sorage requiremen consrained minimizaion Quasi-Newon mehods 0 2-2

3 Quasi-Newon mehods key idea: approximae Hessian marix using only gradien informaion x + = x η H }{{} f(x ) surrogae of ( 2 f(x )) challenges: how o find good approximaion H 0 of ( 2 f(x ) ) using only gradien informaion using limied memory achieving super-linear convergence Quasi-Newon mehods -3

4 Crierion for choosing H Consider following approximae quadraic model of f( ): f (x) := f(x + )+ f(x + ), x x which saisfies f (x) = f(x + ) + H+ ( x x + ) ( x x + ) H ( + x x + ) One reasonable crierion: gradien maching for laes wo ieraes: f (x ) = f(x ) f (x + ) = f(x + ) (.a) (.b) Quasi-Newon mehods -

5 rf (x+ ) + H+ x sar x+ (x ) We wih=a rf monooniciy resul: () Secan equaion rf (x+ ) = rf (x+ ) holds For (), one requires 2.5(x+ ) ()rf H+ x+ Lemma x = rf (xrivially. ) Proof of Lemma 2.5 x = rf (x Le f be convex and L-smooh. rf If (x ) +=H/L, + x hen rf (x) + () H x asú follows x = rf (x+ ) r Consider approximae quadraic model of+ f( ) Îx+ xfú Î ( ) 2 Æ Îx x Î2 + + I follows+ ha ú ú ) rf >(x) is any minimizer wih opimal (x+ f (x) := f (xwhere ) +xhrf (x+ ), x x+ i+ x fx H+ x 2 slope H xú Î22 =.x xú (Òf (x ) Òf (xú )).2 whichîx saisfies + rf (x) = rf (x+ ) + H+ x x =0..2. x+.x xú. 2 Èx xú, Òf (x ) Òf (xú )Í + 2.Òf (x ) Òf (xú 2crierion: One= reasonable gradien maching for laes wo ieraes: (.b) holds auomaically. To saisfy (.a), one requires Gradien mehods 2 2 Ø ÎÒf (x ) Òf (xú )Î2 (smoohness). (x )+. (x+.2. rf rf ) (x ) x= =2(x f.òf +(xx Æ. f x xú.2 ) + 2H ) Òf (xú ) ) = rf (x )..2 rf (x ú. H x+ x = f (x+ ) f (x ) Æ.x x L + {z 2 () holds rivially. For (), one requires secan equaion } + rf (x+ ) + H+ xdisplacemen = rf (x ) secan equaion requires ha maps + x + + x x ino change of gradiens f (x ) f (x ) () H+ x+ x = rf (x+ ) rf (x ) Gradien mehods Quasi-Newon mehods -5

6 Secan equaion H + ( f(x + ) f(x ) ) }{{} :=y = x + x }{{} :=s (.2) only possible when s y > 0, since s y = y H + y > 0 admi infinie soluions, since degrees of freedom O(n 2 ) in choosing H+ far exceeds number of consrains n in (.2) which H+ shall we choose? Quasi-Newon mehods -6

7 Broyden-Flecher-Goldfarb-Shanno (BFGS) mehod Quasi-Newon mehods -7

8 Closedness o H In addiion o secan equaion, choose H + sufficienly close o H : minimize H subjec o H H H = H Hy = s for some norm exploi pas informaion regarding H choosing differen norms resuls in differen quasi-newon mehods Quasi-Newon mehods -8

9 Choice of norm in BFGS Choosing M := W /2 MW /2 F for any weigh marix W obeying W s = y, we ge minimize H subjec o W /2 (H H )W /2 F H = H Hy = s This admis closed-form expression H + = ( I ρ s y ) ( H I ρ y s ) + ρ s s }{{} BFGS updae rule; H + 0 if H 0 (.3) wih ρ = y s Quasi-Newon mehods -9

10 An alernaive inerpreaion H + is also soluion o minimize H H, H log de ( H H ) n }{{} KL divergence beween N (0,H ) and N (0,H ) subjec o Hy = s minimizing KL divergence subjec o secan equaion consrains Quasi-Newon mehods -0

11 BFGS mehods Algorihm. BFGS : for = 0,, do 2: x + = x η H f(x ) (line search o deermine η ) 3: H + = ( I ρ s y ) ( H I ρ y s ) + ρ s s, where s = x + x, y = f(x + ) f(x ), and ρ = y s each ieraion coss O(n 2 ) (in addiion o compuing gradiens) no need o solve linear sysems or inver marices no magic formula for iniializaion; possible choices: approximae inverse Hessian a x 0, or ideniy marix Quasi-Newon mehods -

12 Rank-2 updae on H From Sherman-Morrison-Woodbury formula ( A + UV ) = A A U ( I + V A U ) V A, BFGS rule is equivalen o H+ = H s H H s s H + ρ y y s }{{} rank-2 updae Quasi-Newon mehods -2

13 Local superlinear convergence Theorem. (informal) Suppose f is srongly convex and has Lipschiz-coninuous Hessian. Under mild condiions, BFGS achieves x + x 2 x x = 0 2 lim ieraion complexiy: larger han Newon mehods bu smaller han gradien mehods asympoic resul: holds when Quasi-Newon mehods -3

14 Key observaion BFGS updae rule achieves lim ( H 2 f(x ) )( x + x ) 2 x + x 2 = 0 Implicaions even hough H may no converge o 2 f(x ), i becomes increasingly accurae approximaion of 2 f(x ) along search direcion x + x asympoically, x + x ( 2 f(x ) ) f(x ) } {{ } Newon search direcion Quasi-Newon mehods -

15 rgence analysis 7- Convergence analysis Numerical example definiion of mehods Opimaliy offirs-order Neserov s mehod More precisely, convex and L-smooh funcion f s.. HL = V> V> m HL m V m V Example L L noes > LemmaH5., V> immediae V> m V > EE236C lecure +H arrive V>ma = we m mv V m+ s m s m V m+ Using ae arrive a Theorem 5.3 More precisely, convex and L-smooh funcion f s.. as long as xk œ x0 + span{òf (x0 ),, Òf (xk )} for all Æ k Æ 7- y> s Ineresingly, no firs-order mehods can improve upon Neserov s resul in general as long as xk œ x0 + span{òf (x0 ),, Òf (xk )} for all Æ k Æ wih = wih = BFGS updae rule y > s 3LÎx0 xú Î22 32( + ) ! 2 " 0 9 Quasi-Newon cos permehods Newon ieraion: O(n3) plus compuing r2f (x) Acceleraed GD Acceleraed GD 0 R := supxœc DÏ x, x, hen 2 0A 0 A 0 Ô 50 B 00 B 0 Ô Lf R log k Lf R log k bes, op Ô Ô f f ÆO ÆO Ô Ô fl n = 00, N = 500 fl op definiion of firs-order mehods Ô 9! " 2flR Ô wih 0IfD Ï = supxœc x, xl0 f, hen 2? 6 k 0 (x )) ffop Ø ff(x f (xk ) f? f (x ) f op Ø 3LÎx0 xú Î22 32( + )2 = f > >> m + m V> + VX V m+ Vs> m+ T mm+2 V>s m sv s m+ N m m+ V m+ X minimize c>t x> log(b a x) i > i > > > n + x R x V minimize log b s ai xv m+ V m V c + m+ + m+2 s s i s m+ i= >coninuous i= Suppose f is convex and Lipschiz (i.e. Îg Îú Æ Lf ) on C, + + s s = 00, m = 500 chizn coninuous (i.e. Îg Îú ÆSherman-Morrison-Woodbury L f) on C, From formula, BFGS ru suppoe isnewon fl-srongly convex w.r.. formula, Î Î. Then onvexand w.r.. ÎFrom Î. ÏThen Sherman-Morrison-Woodbury BFGS Newon BFGS rule is equivalen Newon H+ =H! H s s> + y H " > L2f sq 2 > >! 0 0 " Lf q s H H = H H s s H + y y + 2 sup {z 0 0 D x, x + > Ï xœc k=0 k pxœc DÏ x, x + s H s 2fl bes, op k k=0 2fl f {z } rank-2 updae Æ q3 q3 f 0 0 rank-2 updae k k=0 k=0 k 50-5

16 Limied-memory quasi-newon mehods Hessian marices are usually dense. For large-scale problems, even soring (inverse) Hessian marices is prohibiive Insead of soring full Hessian approximaions, one may wan o mainain more parsimonious approximaion of Hessians, using only a few vecors Quasi-Newon mehods -6

17 Limied-memory BFGS (L-BFGS) H + = V H V + ρ s s }{{} BFGS updae rule wih V = I ρ y s key idea: mainain modified version of H implicily by soring m (e.g. 20) mos recen vecor pairs (s, y ) Quasi-Newon mehods -7

18 Limied-memory BFGS (L-BFGS) L-BFGS mainains H L = V V mh L,0V m V + ρ m V V m+s m s mv m+ V + ρ m+ V V m+2s m+ s m+v m+ V + + ρ s s can be compued recursively iniializaion H,0 L may vary from ieraion o ieraion only needs o sore {(s i, y i )} m i< Quasi-Newon mehods -8

19 Reference [] Numerical opimizaion, J. Nocedal, S. Wrigh, [2] Opimizaion mehods for large-scale sysems, EE236C lecure noes, L. Vandenberghe, UCLA. [3] Opimizaion mehods for large-scale machine learning, L. Boou e al., arxiv, 206. [] Convex opimizaion, EE36B lecure noes, S. Boyd, Sanford. Quasi-Newon mehods -9