Lecture 9: September 25

Transcription

1 0-725: Opimizaion Fall 202 Lecure 9: Sepember 25 Lecurer: Geoff Gordon/Ryan Tibshirani Scribes: Xuezhi Wang, Subhodeep Moira, Abhimanu Kumar Noe: LaTeX emplae couresy of UC Berkeley EECS dep. Disclaimer: These noes have no been subjeced o he usual scruiny reserved for formal publicaions. They may be disribued ouside his class only wih he permission of he Insrucor. 9. Review of Generalized Gradien Descen Generalized Gradien Descen is used o solve he following problem, min x f(x) = g(x) + h(x) where g is convex and differeniable, h(x) is convex and no necessarily differeniable. Paricularly, here are some special cases of Generalized Gradien Descen (herefore all have O(/k) convergence rae): h = 0, which is simply he gradien descen. h = I C, which is called projeced gradien descen. The problem is defined as: Given closed, convex se C R n, we wan: which is equivalen o where I C (x) = { 0 x C x C min g(x) x C min g(x) + I C (x) x is he indicaor funcion of C. Hence prox (x) = arg min z 2 x z 2 + I C (z) = arg min x z 2 z C which is P C (x), he projecion operaor ono C. Therefore he generalized gradien updae sep is: x + = P C (x g(x)), which performs usual gradien descen updae and hen projec back o C. This is called projeced gradien descen. Some of he easy-o-projec-ono ses C are:. Affine images C = {Ax + b : x R n } 2. Soluion se of linear sysem C = {x R n : Ax = b} 3. Nonnegaive orhan C = {x R n : x 0} = R n + 4. Norm balls C = {x R n : x p }, for p =, 2, 5. Some simple polyhedra and simple cones However, i s worh o noe ha P C can be very hard even for seemingly simple se C, like i s generally very hard o projec ono soluion se of arbirary linear inequaliies, i.e., arbirary polyhedron C = {x R n : Ax b}. 9-

2 9-2 Lecure 9: Sepember 25 g = 0, which is called proximal minimizaion. Then generalized gradien updae sep is jus a prox updae x + = arg min z 2 x z 2 + h(z) This is faser han subgradien mehod (O(/k) compared o O(/ k)), bu i s no implemenable unless we know prox funcion in a closed form. Some issues regarding he Generalized Gradien Descen mehod: In Generalized Gradien Descen, we assume ha he minimizaion prox (x) = arg min z 2 x z 2 +h(z) can be done exacly, wha if we canno evaluae he prox funcion? In his case, if we jus rea his as anoher minimizaion problem and obain an approximae soluion, all bes are off and he pracical convergence rae can be very slow. Bu here are also some excepions like parial proximaion minimizaion [B94]. In he nex secion we ll be alking abou acceleraion, which you can ge some flavor by looking a Fig 9.. Figure 9.: Comparison of differen gradien mehods 9.2 Acceleraion for composie funcions There are four acceleraion ideas proposed by Neserov (983, 998, 2005, 2007). The firs wo are proposed for smooh funcions. The hird idea is smoohing echniques for nonsmooh funcions, coupled wih original acceleraion idea. The fourh is he acceleraion idea for composie funcions, which requires enire hisory of previous seps and makes wo prox calls in each sep. In 2008 Beck and Teboulle exend Neserov s idea (983) o composie funcions, which uses only informaion from wo las seps and makes one prox call. In his noe we mainly focus on his idea.

3 Lecure 9: Sepember Acceleraed generalized gradien mehod The problem is: min x R n g(x) + h(x) where g(x) is convex and differeniable, h(x) is convex and no necessarily differeniable. The acceleraed generalized gradien descen mehod works in his way: choose any iniial x (0) = x ( ) R n repea for k =, 2, 3,... y = x (k ) + k 2 k + (x(k ) x (k 2) ) x (k) = prox k (y k g(y)) Some noes abou he acceleraed generalized gradien mehod are:. Firs sep k = is jus usual generalized gradien updae: x () = prox (x (0) g(x (0) )) 2. Afer he firs sep, he mehod carries some momenum from previous ieraions 3. h = 0 gives acceleraed gradien mehod 4. The mehod acceleraes more owards he end of ieraions, as shown in Fig 9.2. Figure 9.2: A figure showing he acceleraion coefficien varying wih number of ieraions Here are wo examples (Fig 9.3) showing he performance of acceleraed gradien descen compared wih usual gradien descen Reformulaion of he acceleraed generalized gradien mehod To make he convergence analysis easier, we can reformulae he acceleraed generalized gradien mehod as: Iniialize x (0) = u (0)

4 9-4 Lecure 9: Sepember 25 Figure 9.3: Performance of acceleraed gradien descen compared wih usual gradien descen repea for k =, 2, 3,... y = ( θ k )x (k ) + θ k u (k ) x k = prox k (y k g(y)) u (k) = x (k ) + θ k (x (k) x (k ) ) where θ = 2/(k + ). Noe his reformulaion is equivalen o he acceleraed generalized gradien mehod presened in secion 9.2. since u (k ) = x (k 2) + θ k (x (k ) x (k 2) ), hen we have: y = ( θ k )x (k ) + θ k u (k ) = ( θ k )x (k ) + θ k x (k 2) + θ k θ k (x (k ) x (k 2) ) = x (k ) + ( θ k θ k θ k )(x (k ) x (k 2) ) = x (k ) + k 2 k + (x(k ) x (k 2) ) 9.3 Convergence Analysis Jus as he generalized gradien mehod, we minimize f(x) = g(x) + h(x) assuming ha: g is convex, differeniable, g is Lipschiz wih consan L > 0, and h is convex, he prox funcion can be evaluaed. Theorem 9. Acceleraed generalized gradien mehod wih fixed sep size /L saisfies: f(x (k) ) f(x ) 2 x(0) x 2 (k + ) 2 This heorem ells us ha acceleraed generalized gradien mehod can achieve he opimal O(/k 2 ) rae for firs-order mehod, or equivalenly, if we wan o ge f(x (k) ) f(x ) ɛ, we only need O(/ ɛ) ieraions. Now we prove his heorem. Proof: In he proof we focus on one ieraion and drop k noaion, so x +, u + are updaed versions of x, u. Firs we bound boh g(x + ) and h(x + ).

5 Lecure 9: Sepember Since /L and g is Lipschiz wih consan L > 0, we have g(x + ) g(y) + g(y) T (x + y) + L 2 x+ y 2 g(y) + g(y) T (x + y) + 2 x+ y 2 (9.) Suppose we have v = prox (w) = arg min v 2 w v 2 + h(v) 0 ( 2 w v 2 + h(v)) = (w v) + h(v) (w v) h(v) According o he definiion of subgradien, we have for all z, h(z) h(v) (v w)t (z v) h(v) h(z) + (v w)t (z v) for all z, w and v = prox (w). Since x + = prox (y g(y)), subsiue x + in he above inequaliy we ge for all z, h(x + ) h(z) + (x+ y + g(y)) T (z x + ) = h(z) + (x+ y) T (z x + ) + g(y) T (z x + ) (9.2) Adding inequaion 9. and 9.2 we ge for all z, f(x + ) g(y) + h(z) + (x+ y) T (z x + ) + 2 x+ y 2 + g(y) T (z y) Using g(z) g(y) + g(y) T (z y) since g is convex, we furher ge f(x + ) f(z) + (x+ y) T (z x + ) + 2 x+ y 2 Now ake z = x and z = x and muliply boh sides by ( θ) and θ respecively, ( θ)f(x + ) ( θ)f(x) + θ Adding hese wo inequaliies ogeher, we ge (x + y) T (x x + ) + θ x + y 2 2 θf(x + ) θf(x ) + θ (x+ y) T (x x + ) + θ 2 x+ y 2 f(x + ) f(x ) ( θ)(f(x) f(x )) (x+ y) T (( θ)x + θx x + ) + 2 x+ y 2 (9.3) Using u + = x+ θ (x+ x) and y = ( θ)x+θu, we have ( θ)x+θx x + = θ(x u ) and x + y = θ(u + u), subsiue hese equaions o he RHS of inequaion 9.3 we have f(x + ) f(x ) ( θ)(f(x) f(x )) θ 2 (u+ u) T [2θ(x u + ) + θ(u + u)] Back o k noaion we have: θ 2 k = θ2 2 [(x u) (x u + )] T [(x u + ) + (x u)] = θ2 2 ( x u 2 x u + 2 ) (f(x (k) ) f(x )) + 2 u(k) x 2 ( θ k) θk 2 (f(x (k ) ) f(x )) + 2 u(k ) x 2

6 9-6 Lecure 9: Sepember 25 Using θ k θ 2 k θ 2 k we have (f(x (k) ) f(x )) + 2 u(k) x 2 θ 2 k θ 2 k (f(x (k ) ) f(x )) + 2 u(k ) x 2 Ierae his inequaliy and use θ =, u (0) = x (0) we ge θ 2 k (f(x (k) ) f(x )) + 2 u(k) x 2 ( θ ) θ 2 (f(x (0) ) f(x )) + 2 u(0) x 2 = 2 x(0) x 2 Hence we conclude f(x (k) ) f(x ) θ2 k 2 x(0) x 2 = 2 x(0) x 2 (k + ) Acceleraed Backracking Line Search In he proof for acceleraed general gradien descen we saw an O(/k 2 ) convergence rae which opimal. Gradien descen on he oher hand has a O(/k) convergence rae. The proofs for hese convergence raes are very differen and are made under differen assumpions. There are a number of differen acceleraed backracking schemes and hese are made under differen crieria for he same reason. We will examine one of he simpler schemes. The assumpions ha his scheme needs o make are : Lipschiz Gradien condiion g(x + ) g(y) + g(y) T (x + y) + 2 x+ y 2 Condiion on θ k ( θ k ) k θ 2 k k θ 2 k k is monoonically decreasing i.e. k k. The problem wih his condiion is ha if you choose a small sep size iniially, you will need o coninue o use small sep sizes furher on as well Algorihm Choose β < 0 = for k =, 2, 3,... ill convergence k = k x + = prox k (y k g(y)) while g(x + ) > g(y) + g(y) T (x + y) + 2 k x + y 2 repea k = β k x + = prox k (y k g(y)) endwhile endfor This mehod checks if g(x + ) is small enough, oherwise i shrinks k by a facor β and updaes x +. This mehod saisfies he required condiions and is hus able o achieve a O(/k 2 ) convergence rae.

7 Lecure 9: Sepember Convergence raes From he above discussion, he following heorem summarizes he O(/k 2 ) convergence rae Theorem 9.2 Acceleraed generalized gradien mehod wih backracking saisfies f(x (k) ) f(x ) 2 x(0) x 2 min (k + ) FISTA Fas Ieraive Sof Thresholding Algorihm(FISTA) is an acceleraed version of ISTA which is applied o problems conaining convex differeniable objecives wih L norm such as Lasso. The Lasso problem is defined as : min x 2 y Ax 2 + λ x ISTA soluion This is he soluion by using he normal generalized gradien also known as he ieraive sof hresholding algorihm (ISTA). See Lecure 8. where S λ ( ) is he sof-hresholding operaor x (k) = S λk (x (k ) + k A T (y Ax (k ) )) k =, 2, 3,... x i λ if x i > λ [S λ (x)] i = 0 if λ x i λ x i + λ if x i < λ This is obained by solving he prox funcion prox (x) = arg min z R n 2 x z 2 + λ z = S λ (x) FISTA soluion The acceleraed version involves solving he same prox problem bu wih an addiional vecor added o he inpu o he prox funcion. v = x (k ) + k 2 k + (x(k ) x (k 2) ) x (k) = S λk (v + k A T (y Ax (k ) )) k =, 2, 3,... Here we show wo images comparing he performance of ISTA vs FISTA. We can see ha FISTA clearly beas ISTA by an order of magniude faser.

8 9-8 Lecure 9: Sepember 25 Figure 9.4: Performance of ISTA vs FISTA for Lasso Regression (n=00,p=500) Figure 9.5: Performance of ISTA vs FISTA for Lasso Logisic Regression (n=00,p=500) 9.6 Failure Cases for Acceleraion Acceleraion achieves he opimal O(/k 2 ) convergence rae for gradien based mehods. However, i does no do so under all condiions. In some cases i migh perform similar o non-acceleraed mehods and in some ohers i migh acually hur performance. Some cases are presened wherein acceleraion fails o do well.

9 Lecure 9: Sepember Warm Sars In ieraive algorihms warm saring can be an effecive sraegy o speed up convergence. For e.g. in cross validaion runs for Lasso, one can use he opimal ˆx found in he previous ieraion as a warm sar for he nex ieraion. Le he uning parameers for lasso be λ λ 2... λ r When solving for λ, iniialize x (0) = 0 and record soluion ˆx(λ ). Now, reuse his value such ha when solving for λ j, iniialize x (0) (λ j ) = ˆx(λ j ). I has been observed ha over a fine grid of values, generalized gradien descen can perform jus as well as acceleraed version when using warm sars Marix Compleion In he case of marix compleion, acceleraion and even backracking can hur performance. The marix compleion problem is described in Lecure 8. Briefly, Given a marix A, only some enries (i, j) Ω of which are visible o you, you wan o fill in he res of enries, while keeping he marix low rank. We solve, min X 2 P Ω(A) P Ω (X) 2 F + λ X where X = r i= σ i(x) is he nuclear norm, r is he rank of X and P Ω ( ) is he projecion operaor, [P Ω (X)] ij = { X ij (i, j) Ω 0 (i, j) / Ω he gradien descen updaes, also known as he sof-impue algorihm are X + = S λ (P Ω (A) + P Ω (X)) where S λ ( ) is he marix sof-hresholding operaor which requires he SVD o compue as S λ (X) = UΣ λ V T where (Σ λ ) ii = max{σ ii λ, 0}. Calculaing he SVD can be expensive and can cos upo O(mn 2 ) operaions. This can be expensive for he backracking search mehod since each backracking loop evaluaes he generalized G (X) a various values of and his involves solving he prox funcion. This equaes o calling he SVD several imes and can be slow. The marix compleion problem does no work well wih acceleraion as well, since acceleraion involves changing he argumen passed o he prox funcion. We pass y g(y) insead of x g(x). This can make he marix high rank which makes he compuaion of SVD more expensive. Here is an example where using acceleraion performs worse han sof-impue [M].

10 9-0 Lecure 9: Sepember 25 Figure 9.6: Performance of Sof impue vs Acceleraed grad descen on a small and a big problem References [M] [B94] R. Mazumder and T. Hasie and R. Tibshirani Specral Regularizaion Algorihms for Learning Large Incomplee Marices, The Journal of Machine Learning Research, 20, pp Bersekas and Tseng Parial proximal minimizaion algorihms for convex programming 994.