Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation

Transcription

1 Course Noes for EE7C Spring 018: Convex Opimizaion and Approximaion Insrucor: Moriz Hard Graduae Insrucor: Max Simchowiz Ocober 15, Srong convexiy This lecure inroduces he noion of srong convexiy and combines i wih smoohness o develop he concep of condiion number. While smoohness gave as an upper bound on he second-order erm in Taylor s approximaion, srong convexiy will give us a lower bound. Taking ogeher, hese wo assumpions are quie powerful as hey lead o a much faser convergence rae of he form exp Ω. In words, gradien descen on smooh and srongly convex funcions decreases he error muliplicaively by some facor sricly less han 1 in each ieraion. The echnical par follows he corresponding chaper in Bubeck s ex [Bub15]. 3.1 Reminders Recall ha we had a leas wo definiions apiece for convexiy and smoohness: a general definiion for all funcions and a more compac definiion for wice-differeniable funcions. A funcion f is convex if, for each inpu, here exiss a globally valid linear lower bound on he funcion: f y f x + g xy x. For differeniable funcions, he role of g is played by he gradien. A funcion f is β-smooh if, for each inpu, here exiss a globally valid quadraic upper bound on he funcion, wih finie quadraic parameer β: f y f x + g xy x + β x y. More poeically, a smooh, convex funcion is rapped 1

2 beween a parabola and a line. Since β is covarian wih affine ransformaions, e.g. changes of unis of measuremen, we will frequenly refer o a β-smooh funcion as simply smooh. For wice-differeniable funcions, hese properies admi simple condiions for smoohness in erms of he Hessian, or marix of second parial derivaives. A D funcion f is convex if f x 0 and i is β-smooh if f x βi. We furhermore defined he noion of L-Lipschizness. A funcion f is L-Lipschiz if he amoun ha i sreches is inpus is bounded by L: f x f y L x y. Noe ha for differeniable funcions, β-smoohness is equivalen o β-lipschizness of he gradien. 3. Srong convexiy Wih hese hree conceps, we were able o prove wo error decay raes for gradien descen and is projecive, sochasic, and subgradien flavors. However, hese raes were subsanially slower han wha s observed in cerain seings in pracice. Noing he asymmery beween our linear lower bound from convexiy and our quadraic upper bound from smoohness we inroduce a new, more resriced funcion class by upgrading our lower bound o second order. Definiion 3.1 Srong convexiy. A funcion f : Ω R is α-srongly convex if, for all x, y Ω, he following inequaliy holds for some α > 0: f y f x + gx y x + α x y As wih smoohness, we will ofen shoren α-srongly convex o srongly convex. A srongly convex, smooh funcion is one ha can be squeezed beween wo parabolas. If β-smoohness is a good hing, hen α-convexiy guaranees we don have oo much of a good hing. A wice differeniable funcion is α-srongly convex if f x αi. Once again, noe ha he parameer α changes under affine ransformaions. Convenienly enough, for α-srongly convex, β-smooh funcions, we can define a basisindependen quaniy called he condiion number. Definiion 3. Condiion Number. An α-srongly convex, β-smooh funcion f has condiion number β α. For a posiive-definie quadraic funcion f, his definiion of he condiion number corresponds wih he perhaps more familiar definiion of he condiion number of he marix defining he quadraic.

3 A look back and ahead. The following able summarizes he resuls from he previous lecure and he resuls o be obained in his lecure. In boh, he value ɛ is he difference beween f a some value x compued from he oupus of gradien descen and f calculaed a an opimizer x. Convex Srongly convex Lipschiz ɛ O1/ ɛ O1/ Smooh ɛ O1/ ɛ e Ω Table 1: Bounds on error ɛ as a funcion of number of seps aken for gradien descen applied o various classes of funcions. Since a rae ha is exponenial in erms of he magniude of he error is linear in erms of he bi precision, his rae of convergence is ermed linear. We now move o prove hese raes. 3.3 Convergence rae srongly convex funcions For no good reason we begin wih a convergence bound for srongly convex Lipschiz funcions, in which we obain a O1/ rae of convergence. Theorem 3.3. Assume f : Ω R is α-srongly convex and L-Lipschiz. Le x be an opimizer of f, and le x s be he updaed poin a sep s using projeced gradien descen. Le he max number of ieraions be wih an adapive sep size η s = αs+1, hen f s + 1 x s f x L α + 1 The heorem implies he convergence rae of projeced gradien descen for α-srongly convex funcions is similar o ha of β-smooh funcions wih a bound on error ɛ O1/. In order o prove Theorem 3.3, we need he following proposiion. Proposiion 3.4 Jensen s inequaliy. Assume f : Ω R is a convex funcion and x 1, x,...,, x n, i=1 n γ ix i / i=1 n γ i Ω wih weighs γ i > 0, hen n f i=1 γ i x i i=1 n γ n i=1 γ i f x i i i=1 n γ i For a graphical proof follow his link. Proof of Theorem 3.3. Recall he wo seps updae rule of projeced gradien descen y s+1 = x s η s f x s x s+1 = Π Ω y s+1 3

4 Firs, he proof begins by exploring an upper bound of difference beween funcion values f x s and f x. f x s f x f x s x s x α x s x = 1 η s x s y s+1 x s x α x s x by updae rule = 1 η s x s x + x s y s+1 y s+1 x α x s x by "Fundamenal Theorem of Opimizaion" = 1 η s x s x y s+1 x + η s f x s α x s x by updae rule 1 η s x s x x s+1 x + η s f x s α x s x 1 α η s x s x 1 x s+1 x + η sl η s by Lemma?? by Lipschizness By muliplying s on boh sides and subsiuing he sep size η s by, we ge αs+1 s f x s f x L α + α 4 ss 1 x s x ss + 1 x s+1 x Finally, we can find he upper bound of he funcion value shown in Theorem 3.3 obained using seps projeced gradien descen f s + 1 x s = s + 1 f x s by Proposiion 3.4 s f x + L α + α 4 ss 1 x s x ss + 1 x s+1 x f x + s f x + L α + 1 L α + 1 α x +1 x by elescoping sum This concludes ha solving an opimizaion problem wih a srongly convex objecive 1 funcion wih projeced gradien descen has a convergence rae is of he order +1, which is faser compared o he case purely wih Lipschizness. 4

5 3.4 Convergence rae for smooh and srongly convex funcions Theorem 3.5. Assume f : R n R is α-srongly convex and β-smooh. Le x be an opimizer of f, and le x be he updaed poin a sep using gradien descen wih a consan sep size 1 β, i.e. using he updae rule x +1 = x 1 β f x. Then, x +1 x exp α β x 1 x In order o prove Theorem 3.5, we require use of he following lemma. Lemma 3.6. Assume f as in Theorem 3.5. Then x, y R n and an updae of he form x + = x 1 β f x, Proof of Lemma 3.6. f x + f y f x x y 1 β f x α x y f x + f x + f x f y f x x + x + β x+ x Smoohness + f x x y α x y Srong convexiy = f x x + y + 1 β f x α x y Definiion of x + = f x x y 1 β f x α x y Definiion of x + Now wih Lemma 3.6 we are able o prove Theorem 3.5. Proof of Theorem 3.5. x +1 x = x 1 β f x x = x x β f x x x + 1 β f x 1 α x x β Use of Lemma 3.6 wih y = x, x = x 1 α x1 x β exp α x 1 x β 5

6 We can also prove he same resul for he consrained case using projeced gradien descen. Theorem 3.7. Assume f : Ω R is α-srongly convex and β-smooh. Le x be an opimizer of f, and le x be he updaed poin a sep using projeced gradien descen wih a consan sep size 1 β, i.e. using he updae rule x +1 = Π Ω x 1 β f x where Π Ω is he projecion operaor. Then, x +1 x exp α β x 1 x As in Theorem 3.5, we will require he use of he following Lemma in order o prove Theorem 3.7. Lemma 3.8. Assume f as in Theorem 3.5. Then x, y Ω, define x + Ω as x + = Π Ω x β 1 f x and he funcion g : Ω R as gx = βx x+. Then f x + f y gx x y 1 β gx α x y Proof of Lemma 3.8. The following is given by he Projecion Lemma, for all x, x +, y defined as in Theorem 3.7. f x x + y gx x + y Therefore, following he form of he proof of Lemma 3.6, f x + f x + f x f y f x x + y + 1 β gx α x y gx x + y + 1 β gx α x y = gx x y 1 β gx α x y The proof of Theorem 3.7 is exacly as in Theorem 3.5 afer subsiuing he appropriae projeced gradien descen updae in place of he sandard gradien descen updae, wih Lemma 3.8 used in place of Lemma 3.6. References [Bub15] Sébasien Bubeck. Convex opimizaion: Algorihms and complexiy. Foundaions and Trends in Machine Learning, 83-4:31 357,