arxiv: v1 [math.oc] 26 Sep 2018

Transcription

1 An Iteative Regulaized Inceental Pojected Subgadient Metod fo a Class of Bilevel Optiization Pobles Mostafa Aini 1 and Fazad Yousefian axiv:18915v1 [atoc] 6 Sep 18 Abstact We study a class of bilevel convex optiization pobles wee te goal is to find te iniize of an objective function in te uppe level, aong te set of all optial solutions of an optiization poble in te lowe level A wide ange of pobles in convex optiization can be foulated using tis class An ipotant exaple is te case wee an optiization poble is ill-posed In tis pape, ou inteest lies in addessing te bilevel pobles, wee te lowe level objective is given as a finite su of sepaate nondiffeentiable convex coponent functions Tis is te case in a vaiety of applications in distibuted optiization, suc as lage-scale data pocessing in acine leaning and neual netwos To te best of ou nowledge, tis class of bilevel pobles, wit a finite su in te lowe level, as not been addessed befoe Motivated by tis gap, we develop an iteative egulaized inceental subgadient etod, wee te agents update tei iteates in a cyclic anne using a egulaized subgadient Unde a suitable coice of te egulaization paaete sequence, we establis te convegence of te poposed algoit and deive a ate of O 1/ 5 ε in tes of te lowe level objective function fo an abitay sall ε > We pesent te pefoance of te algoit on a binay text classification poble I INTRODUCTION In tis pape, we conside a class of bilevel optiization pobles as follows iniize x P f subject to x X agin y X f y wee f, : R n R denote te lowe and uppe level objective functions, espectively, and X R n is a constaint set Tis is called te selection poble [8], [] as we ae selecting aong optial solutions of a lowe level poble, one tat iniizes te objective function In paticula, we conside te case wee te lowe level objective function is given as f x i=1 f ix, wee f i : R n R is te it coponent function fo i = 1,, We ae te following basic assuptions Assuption 1 Poble popeties: a Te set X R n is nonepty, copact and convex; also X intdo f do b Te functions f i x fo i = 1,, ae pope, closed, convex, and possibly nondiffeentiable c Te function is stongly convex wit paaete µ > and possibly nondiffeentiable Next, we pesent two instances of te applications of foulation Pf 1 Mostafa Aini and Fazad Yousefian ae wit te Depatent of Industial Engineeing and Manageent, Olaoa State Univesity, Stillwate, OK 7478, USA, oaini@ostateedu, fazadyousefian@ostateedu A Exaple pobles i Constained nonlinea optiization: Conside a constained convex optiization poble given as iniize x subject to q i x, fo i = 1,, x X wee X R n is an easy-to-poject constaint set,,q i : R n R fo all i = 1,, ae convex and possibly nonlinea funcitons Tis poble can be efoulated as P f by setting cf [3] f x i=1 f i x = i=1 ax{,q i x} ii Ill-posed distibuted optiization: An optiization poble is called ill-posed wen it as ultiple optial solutions o it is vey sensitive to data petubations [5] Fo instance, in applications aising in acine leaning, conside te epiical is iniization poble wee te goal is to iniize te total loss i=1 La ix,b i, wee a i is te input, b i is te output of it obseved datu and L is te loss function Fo exaple, in logisitic loss egession, L is eely convex In tese cases, anote citeion suc as spasity ay be taen into account fo te optial solution So, to induce spasity, a seconday objective function is consideed in te given poble Fo instance, te wellnown elastic net egulaization can be used as function Hence, to addess ill-posedness, te following bilevel optiization odel is consideed [8], []: iniize x 1 + µ x subject to x agin y X i=1 La T i y,b i, wee µ > egulates te tade-off between l 1 and l nos B Existing etods Poble Pf, tat is also efeed to as ieacical optiization, is a paticula case of ateatical poga wit genealized equation o equilibiu constaint [13], [15] Tee as been a few appoaces to tacle tis poble Note tat in all appoaces te following iniization poble and its iniize ave been extensively utilized Definition 1: Given a paaete λ >, te egulaized poble coesponding to Pf, is defined as iniize f λ x f x + λx 1 subject to x X

2 Pape Assuptions on lowe level obj function TABLE I: Copaison of Metods fo Solving te Bilevel Optiization Poble P f Assuptions on uppe level obj function [4] convex, locally Lipscitz convex, soot ite egu [3] convex, nonsoot convex, nonsoot ite egu [] convex, Lipscitz [] convex, continuously diffeentiable, Lipscitz stongly convex, continuously diffeentiable stongly convex, soot Metodology Metic Convegence MNG SAM [9] convex, diffeentiable stongly convex, diffeentiable ite egu [9] Tis wo convex, continuously diffeentiable convex, nondiffeentiable finite su fo stongly convex, continuously diffeentiable stongly convex, nondiffeentiable ite egu inceental ite egu f f f f f f f f f f f f f f asyptotic asyptotic asyptotic O 1/ asyptotic O 1/ 1/ 1/6 ε O asyptotic O 1/ asyptotic asyptotic O 1/ 5 ε Also, let xλ denote te unique iniize of tis poble We ay categoize te existing algoits as follows i Exact egulaization: Te egulaization tecnique as been igly used in soe applications suc as signal pocessing wit x = x o x = x 1 [5], [3] Tis tecnique needs a pope paaete λ wic is difficult to deteine in ost of cases To addess tis issue, Mangasaian et al [11], [17] intoduced exact egulaization A solution of poble 1 is called exact wen it is in te set X Te ain dawbac of tis appoac is tat te tesold below wic fo any λ te egulaization 1 is exact, is vey difficult to deteine a pioi see [8] ii Iteative egulaization: In tis appoac, te idea is to develop a single-loop scee wee te egulaization paaete is updated iteatively duing te algoit In [4], an explicit descent algoit is poposed, wee poble 1 is solved as a single-level unconstained poble iteatively In te soot case, te convegence is sown wen =1 λ = and li λ = Fo nonsoot cases, a bundle etod was poposed, wic as a descent step fo te weigted cobination of objective functions in te lowe and uppe levels [3] Anote algoit called ybid steepest descent etod HSDM and its extensions wee developed in [8], [19] Te ain dawbac of all tese wos is tat no coplexity analysis was povided fo te undelying algoti In [9], te ate of O1/ 1/6 ε is deived fo a special case of poble Pf, wee te objective function is and te constaint is solutions of a stocastic vaiational inequality poble Recently, a pial-dual algoit in [9] was offeed tat uses te idea of iteative egulaization Despite a sublinea ate in tes of, te algoit can only be applied to continuously diffeentiable and sall-scale ie, = 1 egies iii Minial no gadient: In [], te inial no gadient MNG etod was developed fo solving poble Pf wit = 1 Te ate of O1/ was deived fo te convegence wit espect to lowe level poble Te ain disadvantage of te MNG etod is tat it is a twoloop scee wee at eac iteation a iniization poble sould be solved iv Sequential aveaging: Te sequential aveaging etod SAM, developed in [7], was eployed in [] fo solving te poble in a oe geneal setting Te poposed etod is poved to ave te ate of convegence of O1/ in tes of te function f Te etod is called Big-SAM Despite tat it is a single-loop scee, sequential aveaging scees equie sootness popeties of te poble and see not to lend teselves to distibuted ipleentations C Main Contibutions Fo oe details on te ain distinctions between ou wo and te existing etods see Table 1 In none of te existing etods, te finite su fo fo te lowe level poble is consideed Te su stuctue is vey apant in pactice wen we ave sepaate objective functions elated to diffeent agents in a distibuted setting Tis is te case fo exaple in acine leaning fo vey lage datasets [6], wee eac f i epesents an agent tat is coopeating wit otes Wen te coplete infoation of all te agents, ie suation of all subgadients is not available, tese agents can be teated distinctly Due to its wide ange of applications in distibuted optiization, finite su poble as been extensively studied Aong popula etods ae inceental subgadient IG [4], [18] and inceental aggegated subgadients IAG [5], [6] fo deteinistic and stocastic aveage gadient SAG [1], SAGA [7] and MISO etods [16] fo stocastic egies Tese algoits ave faste convegence and ae coputationally efficient in lage-scale optiization since a vey less aount of eoy is equied at eac step in ode to stoe only one agent s infoation and subsequently update te iteate based on tat [1] Despite te widespead use of tese fist-ode etods, tey do not addess te bilevel poble P f Motivated by te existing lac in te liteatue and inspied by te advantages of inceental appoaces, in tis pape,

3 we let te lowe level objective function to be a suation of coponents Ten we use te idea of inceental subgadient optiization to addess poble Pf We let functions in bot levels to be nondiffeentiable We ten pove te convegence of ou poposed algoit as well as te O1/ 5 ε ate of convegence Rea 1: An inteesting eseac question tat is eained as a futue diection to ou eseac is if we can establis te convegence of iteative egulaized IAG etod in solving poble Pf o siilaly SAG, SAGA and MISO in stocastic egies Notation Te inne poduct of two vectos x,y R n, is sown as x T y Also, denotes Euclidean no nown as Fo a convex function f wit te doain do f, any vecto g f wit f x + g T f y x f y fo all x,y do f, is called a subgadient of f at x We let f x and x denote te set of all subgadients of functions f and at x Let f be te optial value and X epesent te set of all optial solutions of te lowe level poble in Pf and x sows any eleent of tis set Liewise, x denotes te optial solution of poble Pf, and x λ denotes te optial solution of poble 1 Also, we let P X x denote te Euclidean pojection of vecto x onto te set X Te est of tis pape is oganized as follows In Section II, we pesent te algoit outline Ten, we discuss te convegence analysis in Section III, and deive te convegence ate in Section IV We pesent te nueical esults in Section V, and conclude in Section VI II ALGORITHM OUTLINE In tis section, we intoduce te iteative egulaized inceental pojected subgadient IR-IG fo geneating a sequence tat conveges to te unique optial solution of Pf See Algoit 1 IR-IG etod includes two ain Algoit 1 IR-IG initialization: Set an abitay initial point x X, x = x, and S = γ and pic < 1 fo =,1,,N 1 do Set x, = x and pic γ > and λ > fo i =,1,, 1 do Pic g fi+1 x,i f i+1 x,i and g x,i x,i Update x,i using te following elation: x,i+1 := P X x,i γ g fi+1 x,i + λ g x,i end fo Set x +1 := x, Update S and x using: end fo etun x N S +1 := S + γ +1, x +1 := S x + γ +1 x +1 S +1 3 steps Fist, te agents update tei iteates in an inceental fasion siila to te standad IG etod Tis step taes a cicle aound te all coponents of function f to update te iteate Howeve, Te ain diffeence lies in te seconday objective function, wic is added by a vanising ultiplie λ Second, we do aveaging in ode to acceleate te convegence speed of te algoit Fo tis, we conside a weigted aveage sequence { x } defined as below: x +1 := t= ψ t, x t, wee ψ t, γ t i= γ i, 4 in wic < 1 is a constant, contolling te weigts Note tat 3 in Algoit 1 follows fo te elation 4 by applying induction, see eg, Poposition 3 in [3] III CONVERGENCE ANALYSIS In tis section, ou goal is to sow tat te geneated sequence { x } by Algoit 1 conveges to te unique optial solution of poble Pf see Teoe 1 Rea : a Note tat fo Teoe 316, pg 4 of [1], Assuption 1 iplies tat tee exist constants C f,c R suc tat g fi x C f and g x C fo all i = 1,, and x X, wee g fi x f i x and g x x b Fo Teoe 361, pg 71 of [1], functions f i and ae Lipscitz ove X wit paaetes C f and C, espectively, ie, fo all i = 1,, and x,y X f i x f i y C f x y, x y C x y c Assuption 1a,b iply tat te optial solution set, X, is nonepty Hee, we stat wit a lea wic elps bound te eo of optial solutions of te poble 1 fo two diffeent values of λ We will ae use of tis lea in te convegence analysis Te poof fo tis lea can be done in a sae fasion to tat of Poposition 1 in [9] Lea 1: Let Assuption 1 old Suppose {xλ } be te sequence of te optial solutions of poble 1 wit paaete λ := λ Ten, a xλ x λ C 1 λ 1 µ 1 λ b If λ, ten te sequence {xλ } conveges to te unique optial solution of poble Pf, ie, x To get stated, we also need a ecusive uppe bound on te te x +1 xλ Tis is povided by te following lea and will be used in Poposition 1 to pove te convegence of sequence {x } geneated by te algoit to x Lea A ecusive eo bound: Let Assuption 1 old and < µ λ µ Ten, fo te sequence {x } geneated by Algoit 1 and fo all > we ave x +1 xλ 1 γ λ µ x xλ 1 + 3C γ λ µ 3 1 λ 1 λ + 6 γ C f + λ C, wee x λ is te unique optial solution of poble 1 wit λ := λ

4 Poof: Using and te nonexpansiveness popety of pojection, we ave x,i+1 xλ = P X x,i γ g fi+1 x,i + λ g x,i P X xλ x,i xλ + γ g fi+1 x,i + λ g x,i T γ g fi+1 x,i + λ g x,i x,i x λ By boundedness of subgadients fo Rea a, te definition of subgadient fo f i+1, and te stong convexity of, we obtain x,i+1 xλ x,i xλ + γ C f + λ C γ γ λ x,i xλ γ λ µ x,i xλ = 1 γ λ µ x,i xλ γ + γ f i+1 xλ + λ x λ + γ C f + λ C f i+1 x,i f i+1 xλ f i+1 x,i + λ x,i Taing suation fo bot sides ove i, using x, = x,x, = x +1, and tat γ λ µ >, we obtain 1 i= x,i+1 xλ 1 γ λ µ x x 1 λ + x,i xλ + γ C f + λ 1 C γ f i+1 x,i + λ i= x,i + γ f xλ + λ xλ, 5 wee we used te definition of function f in te second inequality Now by eaanging te tes and adding and subtacting f x + λ x we obtain x +1 xλ 1 γ λ µ 1 γ i= + γ i=1 x x λ + γ C f + λ C f i+1 x,i f i+1 x + λ x,i x f xλ + λ xλ f x λ x }{{} 1 γ λ µ Te1 x x λ + γ C f + λ C 1 + γ f i+1 x,i f i+1 x + λ i= }{{} x,i x, }{{} Te Te3 wee Te1 is used due to optiality of x λ fo f +λ Also, fo Rea b we now tat Te C f x,i x and Te3 C x,i x So, We ave x +1 xλ 1 γ λ µ x xλ + γ C f + λ 1 C + C f + λ C γ x,i x 6 i= Next, we find an uppe bound fo x,i x We ave x,1 x = P X x, γ g f1 x, + λ g x, P X x g γ f1 x, + λ g x, γ C f + λ C Fo i >, in a siila way, we ave x,i+1 x x,i x + γ C f + λ C So fo i =,1,, 1, we ave x,i+1 x i + 1γ C f + λ C i + 1γ Cf + λ C 7 Cobining tis wit 6, we will obtain x +1 xλ 1 γ λ µ x xλ Next, we elate x to x λ 1 We ave + 6 γ C f + λ C 8 x xλ x = xλ x 1 + λ xλ 1 T x x λ 1 x λ 1 x λ + } {{ } Te4 Applying te fact tat a T b a /α +α b fo all a,b R n and α > fo Te4 wen α = /γ λ µ, we obtain x xλ x = xλ x 1 + λ xλ 1 + γ λ µ x xλ 1 + = 1 + γ λ µ Using Lea 1a, we obtain γ λ µ x λ x λ 1 x x λ γ λ µ x xλ 1 + γ λ µ C γ λ µ µ x xλ 1 1 λ 1 Plugging tis inequality into 8 we obtain x +1 xλ 1 γ λ µ 1 + γ λ µ + 1 γ λ µ 1 + γ λ µ + 6 γ C f + λ C λ C 1 λ 1 µ λ x λ x λ 1 x xλ 1

5 Te desied esult is obtained fo < µ λ µ We will ae use of te following esult in Poposition 1 Lea 3 Lea 1, pg 49, []: Let {ν } be a sequence of nonnegative scalas and let {α } and {β } be scala sequences suc tat: ν +1 1 α ν + β fo all, α 1, β, = α =, = β β <, li = α Ten, li ν = Assuption : Assue tat fo all we ave a{γ } and {λ } ae non-inceasing positive sequences wit γ λ µ b = γ λ = c 1 λ 1 = γ λ λ 1 < d = γ < 1 λ 1 eli γ λ λ 1 = f γ li λ = Poposition 1 Convegence of {x }: Conside poble Pf Let Assuption 1 and old and {x } be geneated by Algoit 1 Ten, a li x xλ = 1 b If li λ =, x conveges to te unique optial solution of poble Pf, ie, x Poof: a Conside te esult fo Lea We let ν x xλ 1, α γ λ µ β 3C γ λ µ 3 1 λ γ λ C f + λ C Fo Assuption a,b,c, since {γ } and {λ } ae positive and γ λ /µ, we ave α 1 and β and also =1 α = and =1 β < To sow tat all te necessay assuptions fo Lea 3 ae satisfied, we ave li β α = 6 C µ 4 1 li γ λ + 13 C µ li γ λ λ 1 λ C f γ li µ λ Consideing Assuption e,f, we only need to sow tat li γ λ = Since {λ } is non-inceasing fo all, we ave λ γ /λ γ λ So by Assuption f, β li γ λ = Consequently li α = Now Lea 3 can be applied We ave li ν = li x xλ 1 = b Applying te tiangula inequality, we obtain x x x x λ 1 + x λ 1 x, fo all Fo pat a, x x conveges to zeo Also, fo Lea 1b, we now tat wen λ te sequence {xλ } conveges to te unique optial solution of poble Pf, ie, x Teefoe te esult olds To ave pevious poposition wo, we equie tat sequences {γ } and {λ } satisfy Assuption Below, we povide a set of feasible sequences fo tis assuption Te poof is analogous to tat of Lea 5 in [9] Lea 4: Assue {γ } and {λ } ae sequences suc tat γ = γ +1 a and λ = λ wee a,b,γ +1 b and λ ae positive scalas and γ λ µ If a > b, a > 5 and a + b < 1, ten te sequences {γ } and {λ } satisfy Assuption Te following is a useful lea in poving convegence tat we will apply in Teoe 1 Lea 5 Teoe 6, pg 75 of [1]: Let {u t } R n be a convegent sequence wit te liit point û R n and let {α } be a sequence of positive nubes wee = α = Suppose {v } is given by v t= 1 α tu t / 1 t= α t fo all 1 Ten, li v = û Now, we can illustate ou ultiate goal in tis section wic is sowing te convegence of te sequence { x } geneated by Algoit 1 to x Teoe 1 Convegence of { x }: Conside poble Pf Let Assuption 1 old Also assue {γ } and {λ } ae sequences suc tat γ = γ +1 a and λ = λ wee +1 b a,b,γ and λ ae positive scalas and γ λ µ Let { x } be geneated by Algoit 1 If a > b, a > 5, a + b < 1 and a 1, ten { x } conveges to x Poof: Consideing te given assuptions, by Lea 4 we can see tat Assuption olds We ave x +1 x = ψ t, x t ψ t, x ψ t, x t x, t= t= applying t= ψ t, = 1 fo 4 and te tiangula inequality Now conside te definition of ψ and let α t γt, u t x t x and v +1 t= ψ t, x t x Since a 1 we ave t= α t = t= γ t = t= t + 1 a = Te sequence {λ t } is deceasing to zeo due to b >, So, fo Poposition 1b, u t = x t x conveges to zeo Teefoe, fo û = we can apply Lea 5 and tus, x +1 x conveges to zeo IV RATE ANALYSIS In tis section, we fist find an eo bound wit espect to te optial values of te lowe level function f wic indeed sows te feasibility of te poble Pf Ten, we apply it to deive a convegence ate fo te algoit Lea 6: Conside te sequence { x N } geneated by Algoit 1 Let Assuption 1 old and {γ } and {λ } be positive and non-inceasing sequences Ten, fo all N 1 and z X we ave 1 f x N f γ C f + λ C = = + C f = t= Cf + λ C + M γ λ + M γ 1, = wee M,M ae scalas suc tat x M, x M fo all x X

6 Poof: Siila to elation 5, we can ave x +1 x x x + γ C f + λ C 1 γ f i+1 x,i + λ i= x,i + γ f + λ x x x + γ C f + λ C 1 γ f i+1 x,i + γ f + 4γ λ M i= Adding and subtacting f x, we obtain x +1 x x x + γ C f + λ C 1 γ fi+1 x,i f i x + γ f f x + 4γ λ M i= x x + γ C f + λ C 1 + γ f i+1 x,i f i x + γ f f x + 4γ λ M i= Applying Rea b, f i+1 x,i f i x C f x,i x, we ave x +1 x x x + γ C f + λ C 1 + C f γ x,i x + γ f f x + 4γ λ M i= Using te inequality 7, we obtain x +1 x x x γ f f x + γ C f + λ C + C f γ Cf + λ C + 4γ λ M 9 Multiplying bot sides by γ 1 and adding and subtacting γ 1 1 x x to te left and side, we ave γ 1 x +1 x γ 1 1 x x + γ 1 1 γ 1 x x γ f f x + C f + λ C + C f γ +1 Cf + λ C + 4γ λ M Since {γ } in non-inceasing and < 1 we ave γ 1 γ 1 1 Also, by te tiangle inequality x x x + x 4M So, we obtain γ 1 x +1 x γ 1 1 x x + 4M γ 1 1 γ 1 γ f f x + C f + λ C + C f γ +1 Cf + λ C + 4γ λ M Taing suation ove = 1,,,N 1, we obtain γ 1 x N x γ 1 x 1 x + 4M γ 1 γ 1 =1 + C f =1 γ f f x + =1 C f + λ C Cf + λ C + 4M γ λ =1 Reoving non-negative tes fo te left-and side of te peceding inequality, we ave γ 1 x 1 x 4M γ 1 γ f f x + =1 =1 C f + λ C + C f =1 γ +1 Cf + λ C + 4M γ λ 1 =1 Fo 9 fo =, we obtain x 1 x γ f f x + γ C f + λc + C f γ Cf + λ C + 4γ λ M + 4M By ultiplying bot sides of te peceding inequality by γ 1 and suing it wit te elation 1, we obtain 4M γ 1 γ f f x + 4M γ λ = + = = C f + λ C + C f Reaanging te tes we ave = γ f x f + C f = = Now we divide bot sides by of ψ, in 4, = ψ, f x f 1 γ = + C f = = C f + λ C Cf + λ C Cf + λ C + 4M γ λ + 4M γ 1 = = = γ C f + λ C and use te definition Cf + λ C + M γ λ + M γ 1 = We now = ψ, = 1 also f x N = ψ, f x because of convexity of f So, we obtain te desied esult Te following lea, will be used to find a convegence ate stateent in Teoe Lea 7 Lea 9, page 418 in [9]: Fo any scala α 1 and integes l and N wee l N 1, we ave N α+1 l + 1 α+1 α + 1 =l + 1 α l + 1 α + N + 1α+1 l + 1 α+1 α + 1 In te following teoe, we pesent a ate stateent fo Algoit 1

7 Teoe A ate stateent fo Algoit 1: Assue { x N } is geneated by Algoit 1 to solve poble Pf Let Assuption 1 and old and also < ε < 5 and < 1 be abitay constants Assue fo < ε < 5, {γ } and {λ } ae sequences defined as γ = γ ε and λ λ = ε, suc tat γ and λ ae positive scalas and γ λ µ Ten, a Te sequence { x N } conveges to te unique optial solution of poble P f, ie, x b f x N conveges to f wit te ate O 1/N 5 ε Poof: Tougout, we set a := 5+5ε, b := 5 ε a Fo te values of a and b, and tat < 1 and < ε < 5, we ave a > b >, a > 5, a + b = 1 5ε < 1, a = 51 + ε < 515 = 75 < 1 Tis iplies tat all conditions of Teoe 1 ae satisfied Teefoe, { x N } conveges to x alost suely b Since {λ } is a non-inceasing sequence fo Lea 6 we ave 1 f x N f γ C f + λc γ +1 = = + C f Cf + λ C = + M λ γ + M γ 1 = We ave γ = γ / + 1 a and λ = λ / + 1 b, tus Let us define We ave Te1 = Te = = = Te1 Na1 N 1 a 1 1 a Te = a N a1, a = 1 ana1 N 1 a a+b = O N 1 a, N+1 1 a b 1 1 a b + 1 = 1 a N a b 1 N 1 a 1 1 a bn 1 a 1 1 a + 1 a N 1 a 1 = O N b + O So, we ave f x N f O N 1 a N in{1 a,1 a,b} = O N in{1 a,b}, wee we used 1 a 1 a Replacing a and b by tei values, we ave f x N f O N in{5 5ε,5 ε} = O N 5 ε 1, 1 1, 1 Log of aveage loss - - x = 1 1 n x = n x = 1 1 n = =-5 = Iteation f x N f 1 γ = + 1 a C f + λ C γ +1 = + 1 a+1 1, C f Cf + λ C = +M γ 1 N a1 + 1 a+1 + M Reaanging te tes, we ave 1 f x N f γ = + 1 a λ γ M + 1 a+b + M γ 1 = N a1 + C f Cf + λ C + C f + λc = = λ γ + 1 a+b + 1 a Fig 1: IR-IG pefoane fo binay text classification poble wit espect to diffeent initial point x, paaetes γ,λ and Te vetical axis epesents te log of aveage loss and te oizontal axis is te nube of iteations V NUMERICAL RESULTS In tis section, we apply te IR-IG etod on a text classification poble In tis poble we assue to ave a suation of inge loss function, ie, L x,a,b ax{,1 b x,a } fo any saple a,b in te lowe level of P f Te set of obsevations a i,b i ae deived fo te Reutes Copus Volue I RCV1 dataset see [14] Tis dataset as categoized Reutes aticles, fo 1996 to 1997, into fou goups: Copoate/Industial, Econoics, Govenent/Social and Maets In tis application, we use a subset of tis dataset wit N = 5, aticles and 138,91 toens to pefo a binay classification of aticles only wit

8 espect to te Maets class Eac vecto a i epesents te existence of all toens in aticle i, and b i sows wete te aticle belongs to te Maets class To decide tat a new aticle can be placed in te Maets class, te poble in te lowe level sould be solved suc tat te optial solution is a weigt vecto of te toens egading te Maets class wic iniizes te total loss Howeve, to ae suc a decision, an optial solution wit a lage nube of nonzeo coponents ae undesiable To induce spasity, we conside te following bilevel poble: iniize x µ x + x 1 11 subject to x ag in y R n i=1 N/ L a T i 1N/+ j y,b i 1N/+ j, j=1 wee, we let µ = 1 and we conside eac batc of N/ = 1 aticles to be one coponent function wit te total nube of coponent functions = 5 Te function is stongly convex wit paaete µ We let γ and λ be given by te ules in Teoe We study te sensitivity of te etod by canging x, γ, λ, and te aveaging paaete < 1 We finally epot te logait of aveage of te loss function L Te plots in Fig 1 sow te convegence of te IR-IG etod fo te poble 11 Te esults sow te convegece of Algoit 1 wit diffeent initial values suc as te stating point o paaetes γ,λ wile wen we pic a salle te algoit is faste in all te cases VI CONCLUDING REMARKS Motivated by te applications of inceental gadient scees in distibuted optiization, especially in acine leaning and lage data taining, we develop an iteative egulaized inceental fist-ode etod, called IR-IG, fo solving a class of bilevel convex optiization We pove te convegence of IR-IG and establis te coesponding ate in tes of te lowe level objective function We finally apply IR-IG to a binay text classification poble and deonstate te pefoance of te poposed algoit REFERENCES [1] A Bec, Fist-ode etods in optiization, MOS-SIAM Seies on Optiization, Society fo Industial and Applied Mateatics SIAM, Piladelpia, PA, 17 [] A Bec and S Sabac, A fist-ode etod fo finding inial no-lie solutions of convex optiization pobles, Mateatical Pogaing, , 56 [3] A Bec and M Teboulle, A fast iteative sinage-tesolding algoit fo linea invese pobles, SIAM jounal on iaging sciences, 1 9, 183- [4] D Betseas, Nonlinea pogaing, Atena Scientific, Belont, MA, 1999 [5] D Blatt, A Heo, and H Gaucan, A convegent inceental gadient etod wit a constant step-size, SIAM Jounal Of Optiization, 18 7, 9-51 [6] L Bottou and Y Le Cun, On-line leaning fo vey lage data sets, Applied Stocastic Models in Business and Industy, 1 5, [7] A Defazio, F Bac, and S Lacoste-Julien, SAGA: A fast inceental gadient etod wit suppot fo non-stongly convex coposite objectives, in Advances in Neual Infoation Pocessing Systes, MIT Pess, Cabidge, MA, 14, [8] M P Fiedlande and P Tseng, Exact egulaization of convex pogas, SIAM Jounal Of Optiization, 184 7, [9] G Gaigos, L Rosasco, and S Villa, Iteative egulaization via dual diagonal descent, Jounal of Mateatical Iaging and Vision 6 18, 1895 [1] M Gübüzbalaban, A Ozdagla, and P A Paillo, On te convegence ate of inceental aggegated gadient algoits, SIAM Jounal Of Optiization, 7 18, [11] S -P Han and O L Mangasaian, Exact penalty functions in nonlinea pogaing, , [1] K Knopp, Teoy and applications of infinite seies, Blacie & Son Ltd, Glasgow, Geat Bitain, 1951 [13] M Kočvaa and J V Outata, Optiization pobles wit equilibiu constaints and tei nueical solution, Mateatical Pogaing, 111 4, [14] D D Lewis, Y Yang, T G Rose, and F Li, RCV1: A new benca collection fo text categoization eseac, Jounal of Macine Leaning Reseac, 5 4, [15] Z-Q Luo, J-S Pang and D Ralp, Mateatical pogas wit equilibiu constaints, Cabidge Univesity Pess, Cabidge 1996 [16] J Maial, Optiization wit fist-ode suogate functions, in ICML, JMLR Poceedings 8, 13, [17] O L Mangasaian, Sufficiency Of exact penalty iniization, SIAM Jounal on Contol and Optiization, , 37 [18] A Nedić and D Betseas, Inceental subgadient etods fo nondiffeentiable optiization, SIAM Jounal of Optiization, 11 1, 1938 [19] E S H Neto and Á A R De Pieo, On petubed steepest descent etods wit inexact line seac fo bilevel convex optiization, Optiization, 6 11, 9918 [] B T Polya, Intoduction to optiization, Optiization Softwae, Inc, New Yo, 1987 [1] N L Roux, M Scidt, and F R Bac, A stocastic gadient etod wit an exponential convegence ate fo finite taining sets, in Advances in Neual Infoation Pocessing Systes 5, F Peeia, C J C Buges, L Bottou, and K Q Weinbege, eds, MIT Pess, Cabidge, MA, 1, [] S Sabac and S Sten, A fist-ode etod fo solving convex bilevel optiization Pobles, SIAM Jounal on Optiization, 7 17, [3] M Solodov, A bundle etod fo a class of bilevel nonsoot convex iniization pobles, SIAM Jounal Of Optiization, 181 7, 4-59 [4] M Solodov, An explicit descent etod fo bilevel convex optiization, Jounal of Convex Analysis, 14 7 [5] A N Tionov and V Y Asenin, Solutions of ill-posed pobles, V H Winston and Sons, Wasington, D C, 1977 Tanslated fo Russian [6] P Tseng and S Yun, Inceentally updated gadient etods fo constained and egulaized optiization, Jounal of Optiization Teoy and Applications, 16 14, [7] H-K Xu, Viscosity appoxiation etods fo nonexpansive appings, Jounal of Mateatical Analysis and Applications, 981 4, [8] I Yaada, M Yuawa, and M Yaagisi, Miniizing te Moeau envelope of nonsoot convex functions ove te fixed point set of cetain quasi-nonexpansive appings, in Fixed-Point Algoits fo Invese Pobles in Science and Engineeing, Spinge, New Yo, 11, 3459 [9] F Yousefian, A Nedić, and U V Sanbag, On Sooting, egulaization and aveaging in stocastic appoxiation etods fo stocastic vaiational inequality pobles, Mateatical Pogaing, , [3] F Yousefian, A Nedić, and U V Sanbag, On stocastic iopox algoits fo stocastic Catesian vaiational inequalities: andoized bloc coodinate, and optial aveaging scees, Set-Valued and Vaiational Analysis, 18, ttps://doiog/117/s