A Forward-Backward Splitting Method with Component-wise Lazy Evaluation for Online Structured Convex Optimization

Transcription

1 A Forward-Backward Spliing Mehod wih Componen-wise Lazy Evaluaion for Online Srucured Convex Opimizaion Yukihiro Togari and Nobuo Yamashia March 28, 2016 Absrac: We consider large-scale opimizaion problems whose objecive funcions are expressed as sums of convex funcions. Such problems appear in various fields, such as saisics and machine learning. One mehod for solving such problems is he online gradien mehod, which explois he gradiens of a few of he funcions in he objecive funcion a each ieraion. In his paper, we focus on he sparsiy of wo vecors, namely a soluion and a gradien of each funcion in he objecive funcion. In order o obain a sparse soluion, we usually add he L1 regularizaion erm o he objecive funcion. Then, because he L1 regularizaion erm consiss of all decision variables, he usual online gradien mehods updae all variables even if each gradien is sparse. To reduce he number of compuaions required for all variables a each ieraion, we can employ a lazy evaluaion. This only updaes he variables corresponding o nonzero componens of he gradien, by ignoring he L1 regularizaion erms, and evaluaes he ignored erms laer. Because a lazy evaluaion does no exploi he informaion relaed o all erms a every ieraion, he online algorihm using lazy evaluaion may converge slowly. In his paper, we describe a forward-backward spliing ype mehod, which explois he L1 erms corresponding o he nonzero componens of he gradien a each ieraion. We show ha is regre bound is O( T ), where T is he number of funcions in he objecive funcion. Moreover, we presen some numerical experimens using he proposed mehod, which demonsrae ha he proposed mehod gives promising resuls. Keywords: online opimizaion; regre analysis; ex classificaion 1 Inroducion In his paper, we consider he following problem: min { f (x) + r(x), (1) 1 Technical Repor , March 27, Graduae School of Informaics, Kyoo Universiy, Yoshida-honmachi, Kyoo, , Japan. nobuo@i.kyoo-u.ac.jp 1

2 where f : R n R( 1,..., T ) are differeniable convex funcions, and r : R n R is a coninuous convex funcion ha is no necessarily differeniable. Typical examples of his problem appear in machine learning and signal processing. In hese examples, f is regarded as a loss funcion, such as a logisic loss funcion f (x) log(1 + exp( b a, x )) [2, 7] or a hinge loss funcion f (x) [ b a, x + 1] + [4]. Here, (a, b ) R n+1 is a pair from he h sample daa. Furhermore, r is a regularizaion funcion o avoid over-fiing, such as he L1 regularizaion erm r(x) λ n x(i) or he L2 regularizaion erm r(x) λ 2 x 2. For some applicaions, his may be an indicaor funcion of he feasible se Ω, which is defined by { 0 if x Ω r Ω (x) + oherwise. In his paper, we consider problem (1) under he following circumsances: (i) Eiher he upper limi of he summaion T is very large, or he funcions f are given one by one a he h ieraion; ha is, he number of funcions f increases wih. { (ii) The gradien f (x) is sparse; ha is, I n, where I i f (x) 0. (iii) The funcion r is decomposable. In machine learning and saisics, T corresponds o he number of raining samples, and f is a loss funcion for he h sample. Therefore, if we have large number of samples, T becomes large. When f is he logisic funcion given above and some sample daa a in f is sparse, hen he gradien of f is also sparse. Furher, if r is he L1 or L2 regularizaion erm, hen r can be decomposed as r(x) ), (2) where r i : R R and denoes he ih componen of he decision variable x. Throughou his paper, we assume ha r is expressed as (2). We can easily exend he subsequen discussions o he case where r is decomposable wih respec o blocks of variables, such as he group lasso. For problems saisfying (i)-(iii), he online gradien mehod is useful. This generaes a sequence of reasonable soluions using only he gradien of f a he h ieraion [15, 16]. Noe ha his is essenially he same as he incremenal gradien mehod [1] and he sochasic gradien mehod [5, 6]. In his paper, we adop he online gradien mehod for he problem (1) saisfying (i)-(iii). We usually evaluae he online gradien mehod using a regre, which is defined by R(T ) 1 { f (x ) + r(x ) min x { f (x) + r(x), (3) where {x is a sequence generaed by he online gradien mehod. The regre R(T ) is he residual beween he opimum value of (1) and he sum of he values of f a x. When R(T ) R(T ) saisfies lim T T 0, we conclude ha he online gradien mehod converges o an opimal soluion. 1 2

3 Various online gradien mehods have been proposed for problem (1) [14]. When he objecive funcion consiss of non-differeniable funcions, such as he L1 regularizaion erm, he online subgradien mehod for problem (1) updaes a sequence as follows [5]: x +1 x α ( f (x ) + η ), where η r(x ) and α is he sepsize a he h ieraion. If f is convex, hen he regre of he online subgradien mehod is O( T ). Furhermore, if f is srongly convex, hen i reduces o O(log T ) [14]. However, his algorihm does no fully exploi he srucure of r, and i may converge slowly. The forward-backward spliing (FOBOS) mehod [8, 9] uses he srucure of r. This consiss of he gradien mehod for f and he proximal poin mehod for r; ha is, a sequence is generaed as { x +1 argmin x f (x ), x + 1 x x 2 + r(x). (4) 2α For given f (x), we can compue (4) in O( I ) when r i (x) 0 for all i. Therefore, when f (x) is sparse as described in (ii), such ha I n, we can obain x +1 quickly. However, when r i (x) 0 for all i we require a compuaion ime of O(n) for (4), even if I n. In order o overcome his drawback, Langford [10] proposed he lazy evaluaion mehod, which defers he evaluaion of r in (4) o a laer round a mos ieraions, and explois he informaion relaing o each deferred r all ogeher afer a number of ieraions. This echnique enables us o obain x +1 using (4) wih only a few compuaions when f (x) is sparse. However, his mehod does no fully exploi he informaion of r a all ieraions, and hence is behavior is inferior. Furhermore, even if we employ he L1 regularizaion erm as r, x canno be sparse excep a he ieraion where he deferred evaluaions of r are performed. In his paper, we propose a novel algorihm ha conducs he lazy evaluaion for r i wih respec o i such ha f (x ) 0 a each ieraion. The usual lazy evaluaion process ignores r for almos all ieraions, and evaluaes r a a few ieraions while updaing all componens in x. In conras, he proposed mehod conducs he lazy evaluaion for he updaed variables +1 (i I ) a each ieraion. Thus, he proposed mehod can exploi he full informaion given by r i (i I ) a each ieraion, and hence he FOBOS mehod is expeced o converge faser wih his echnique han wih he usual lazy evaluaion [10]. The remainder of his paper is organized as follows. In Secion 2, we inroduce some online gradien mehods and he exising lazy evaluaion procedure. In Secion 3, we propose a FOBOS mehod wih componen-wise lazy evaluaion. In Secion 4, we prove ha he regre of he proposed mehod is O( T ). In Secion 5, we presen some numerical experimens using he proposed mehod. Finally, we provide some concluding remarks in Secion 6. Throughou his paper, we employ he following noaions. Noe ha x and x p denoe he Euclidean and p-norm of x, respecively. For a differeniable funcion f : R n R, (i) f(x) denoes he ih componen of f(x). Furhermore, f denoes he subgradien of f. The funcion [ ] + denoes he max funcion; ha is, [a] + max{a, 0. We are aware of ha Lipon and Elkan [11] quie recenly presened similar idea for he problem. The idea of he proposed mehod in his paper were presened in he bachelor hesis of one of he auhors of his paper [12]. Furher, he regre analysis and numerical experimens are presened a he meeing of Operaions Research Sociey of Japan [13], which has been held in March,

4 2 Preliminaries In his secion, we inroduce he exising online gradien mehods, which are he basis of he mehod ha will be proposed in Secion Forward-Backward Spliing Mehod In his subsecion, we inroduce some well-known online gradien mehods for solving problem (1). The mos fundamenal algorihm for (1) is he online subgradien mehod [5], given as follows. Algorihm 1 Online Subgradien Mehod Sep 0: Le x 1 R n and 1. Sep 1: Compue he subgradien s {f (x ) + r(x ). Sep 2: Updae he sequence by x +1 argmin x { s, x + 1 2α x x 2. Sep 3: Updae he sepsize. Se + 1, and reurn o Sep 1. When all of he loss funcions f are convex and α c, he regre (3) of he online subgradien mehod is O( T ) [16], where c is a posiive consan. Moreover, when he funcions f are srongly convex, he regre reduces o O(log T ) [14]. However, his algorihm is inefficien when r is a non-differeniable funcion, such as he L1 regularizaion erm. The forward-backward spliing (FOBOS) mehod [8] is an efficien algorihm for problems where r has a special srucure. Algorihm 2 Forward-Backward Spliing Mehod (FOBOS) Sep 0: Le x 1 R n and 1. Sep 1: Se I {i (i) f (x ) 0 and updae he sequence using he gradien mehod. { { ˆ +1 argminy α (i) f (x )y (y x(i) ) 2 if i I (5) oherwise. Sep 2: Evaluae r i by he proximal poin mehod. { +1 argmin y r i (y) + 1 (y ˆ +1 2α )2 for i 1,..., n. (6) Se + 1, and reurn o Sep 1. When ) λ for a posiive consan λ, (6) is rewrien as +1 sgn( α (i) f (x ) ) max { α (i) f (x ) λα, 0 for i 1,..., n, 4

5 where sgn( ) denoes he signaure funcion; ha is, +1 if a > 0 sgn(a) 0 if a 0 1 oherwise. Moreover, when ) λ 2 x(i)2, (6) is given by λα α 1 + λα (i) f (x ) for i 1,..., n. The regre obained by FOBOS has been demonsraed o be O( T ). We will assume ha condiion (ii) from he inroducion holds. Then, we can noe ha x i +1 xi holds in Sep 1 for i such ha f (x ) 0. Therefore, Sep 1 only updaes for i I. However, even if f (x ) is sparse, Sep 2 has o updae all componens when ) 0 for all i. 2.2 Lazy Evaluaions In his subsecion, we inroduce he lazy evaluaion echnique [10], which ignores (6) in FOBOS for some consecuive ieraions. We carry ou he ignored updaes (6) all ogeher afer every K ieraions. Algorihm 3 FOBOS wih Lazy evaluaion (L-FOBOS) Sep 0: Le K be a naural number, x 1 R n, and 1. Sep 1: Se I {i (i) f (x ) 0 and updae using he gradien mehod. { { argminx α (i) f (x )x (x x(i) ) 2 if +1 If mod K 0, hen se + 1 and reurn o Sep 1. Oherwise, go o Sep 2. Sep 2: Conduc he lazy evaluaion for each r i. Le y 1 x and j 1. i I oherwise. y (i) j+1 argmin y{r i 1 (y) + (y y (i) j ) 2 for i 1,..., n. (7) 2α K+j If j j + 1 and j K, hen reurn o Sep 2. Oherwise, se + 1 and x y j and go o Sep 1. In his paper, we refer o he above algorihm as L-FOBOS. Sep 1 of L-FOBOS updaes only x i +1 for i I using he gradien mehod. Afer K consecuive ieraions of Sep 1, Sep 2 evaluaes he deferred r. Here, Sep 2 employs he same sepsizes as he mos recen K ieraions in Sep 1. If Sep 2 direcly calculaes (7) K imes, hen O(Kn) compuaions are required. However, if r saisfies he following propery, hen he number of compuaions required in Sep 2 can be shown o be O(n). 5

6 Propery 1. Le x 0 R n, and le {x be he sequence obained by he following formula: { 1 x +1 argmin x 2 x x 2 + α r(x). Moreover, le x be he opimal soluion of he following opimizaion problem: { ( K x 1 argmin x 2 x x α )r(x). 1 Then, x K x holds for any x 0. Noe ha he L1 and L2 regularizaion erms and he indicaor funcion for boxed consrains saisfy he above propery [8, Proposiion 11]. When r saisfies Propery 1, we can aggregae all of he compuaions needed for K calculaions (7) ino he following O(n) formula: y (i) K argmin { y r i (y) + 1 (y ) 2 (i 1,..., n), 2 α where α is he sum of he sepsizes of he las K ieraions a he h ieraion; ha is, α j K+1 α j, wih α 0 0. By using L-FOBOS, we can updae he sequence effecively wih respec o he compuaional complexiy. However, he convergence of L-FOBOS migh be slower han FOBOS for many problems, because we canno sufficienly exploi he informaion given by r. Moreover, we canno obain a sparse sequence, even if we se r as he L1 regularizaion erm. Hence, in he nex secion we will presen a novel lazy evaluaion echnique ha can exploi he informaion given by r. 3 FOBOS wih Componen-wise Lazy Evaluaion As discussed in Secion 2, applying he proximal poin mehod o i / I is inefficien for a problem wih sparse gradiens. I may no be useful o updae he variables wihou r, as in he L-FOBOS mehod. Therefore, we propose he following mehod, which explois boh he sparsiy of gradiens and he decomposabiliy of r. Noe ha he proposed algorihm adops he lazy evaluaion of r i for. (i I ) afer he gradien mehod has been used o updae Algorihm 4 is hereafer referred o as CL-FOBOS, where (i) denoes he number of ieraions o elapse since he ieraion where was mos recenly updaed. Noe ha L-FOBOS only updaes once every K ieraions using he informaion of r [10], while CL-FOBOS explois r i (i I ) a each ieraion. Sep 2 of CL-FOBOS requires (i) ieraions for i I. However, we can aggregae all of he necessary compuaions for Sep 2 if r saisfies Propery 1 from Secion 2. From his poin on, we assume ha r saisfies he following assumpion: Assumpion 1. (i) The funcion r is decomposable wih respec o each componen of he decision valuable x. 6

7 Algorihm 4 FOBOS wih Componen-wise Lazy Evaluaion (CL-FOBOS) Sep 0: Le x 1 R n and 1. Sep 1: Se I {i (i) f (x ) 0 and perform he gradien mehod. { { ˆ +1 argminy α (i) f (x )y (y x(i) ) 2 if i I oherwise. (8) Sep 2: Conduc he lazy evaluaion of r i for i I. Le y (i) 1 ˆ and j (i) 1 for i I. y (i) j (i) +1 argmin y{r i 1 (y) + (y y (i) ) 2 for i I 2α (i) +j (i) j (i). Se j (i) j (i) + 1. If j (i) (i), hen reurn o Sep 2. If j (i) > (i), hen se i / I. Furhermore, if I, hen se + 1 and y (i), and go o Sep 1. Oherwise, j (i) reurn o Sep 2. (ii) The funcion r saisfies Propery 1. Under Assumpion 1, Sep 2 can be rewrien as { { 1 argminy 2 (y ˆx(i) ) 2 + (ˇα α (i) +1 )ri (y) if i I ˆ +1 oherwise, ˇα +1 ˇα + α, (9) α (i) ˇα +1 for i I, where ˇα is he sum of he sepsizes from he firs ieraion o he h ieraion, wih ˇα 0 0. Then, ˇα α (i) denoes he sum of he sepsizes from he mos recen ieraion where x i has been updaed o he h ieraion. We see ha he compuaion of α (i) only requires O( I ) compuaions a each ieraion. Afer we calculae f (x ), we updae x in O( I ). 4 Regre Bound of CL-FOBOS In his secion, we derive he regre bound of CL-FOBOS. We can rewrie (8) and (9) as ˆ +1 { +1 { argmin α (i) f (x ) (x(i) ) 2 if i I oherwise, { argmin 1 y R 2 (y ˆx(i) +1 )2 + ( α j (i) +1 j ) r i (y) if i I ˆ +1 oherwise, (10) where (i) denoes he mos recen ieraion where has been updaed prior o he h 7

8 ieraion. The regre bound of CL-FOBOS, R cl (T ), can hen wrien as R cl (T ) { f (x ) + 1 ) { f (x + 1, (11) where x denoes he opimal soluion of problem (1). We will analyze he upper bound of R R cl (T ), and will show ha lim cl (T ) T T 0 holds under he following assumpion. Assumpion 2. (i) (i) K holds, where K is a posiive consan. (ii) There exiss a posiive consan D such ha x x D for 1,..., T. (iii) There exiss a posiive consan G such ha sup vf f (x ) v f G and sup vr r(x ) v r G for 1,..., T. (iv) The sepsize α is given as α α 0, where α 0 is a posiive consan. Assumpion 2 (i) ensures ha all of he componens of {x are updaed a leas once in every K ieraions. The condiions (ii), (iii), and (iv) of Assumpion 2 are assumed in (5) and (6) of he regre analysis of FOBOS. In paricular, Assumpion 2 (ii) implies he boundedness of {x, and hus { ) and { ) are also bounded. Hence, here exis posiive consans R and S such ha ) R for 1,..., T. (12) ) S for i 1,..., n, and 1,..., T. (13) Finally, he sepsize given in Assumpion 2 (iv) is ofen employed for online gradien mehods. Under Assumpions 1 and 2, we now provide a regre bound for CL-FOBOS. Theorem 1. Le {x be a sequence generaed by CL-FOBOS, and suppose ha Assumpions 1 and 2 hold. Then, he regre bound of CL-FOBOS is given by { { R cl (T ) f (x ) + ) f (x + 1 O(log T ) + O( T ). To prove Theorem 1, we use he exising resul on he regre bound of FOBOS. Specifically, we consider he sequence {x given by CL-FOBOS (10) as ha generaed by FOBOS for a cerain convex opimizaion problem. We firs see ha (10) is equivalen o +1 argmin { α (i) f (x ) + α 1 ) (x(i) ) 2, (14) 8

9 where r i : R R is defined as r(y) i c (i) r i (y), c (i) j (i) α +1 j α if i I 0 oherwise. Noe ha (14) generaes he same sequence as (10). The updae (14) consiues he FOBOS procedure for he following problem, wih f and n ri : { min f (x) + r(x) i. (15) 1 Noe ha r i in problem (15) depends on, whereas r(x) of problem (1) does no. Hereafer, we refer o (14) as Algorihm γ. Because Algorihm γ consiues he FOBOS procedure for problem (15), [8] demonsraes ha is regre (16) is given by R γ (T ) {( f (x ) + 1 {( f (x ) + 1 ) ( r(x i (i) ) f (x + ) ( c (i) ) f (x + ) r(x i (i) ) c (i) U γ (T ), (16) ( ) where U γ (T ) 2GD + D 2 T 2α 0 + 7G 2 α 0. Noe ha Rcl (T ) is differen from R γ (T ), whereas {x in R cl (T ) is same as ha in R γ (T ). In fac, c (i) in r i influences R γ (T ). We will firs evaluae he upper bound of R cl (T ) by using R γ (T ). From a simple rearranging, we obain he upper bound of R cl (T ) as follows: where R cl (T ) { f (x ) + 1 {( 1 + f (x ) + i I 1 are consans defined by ) ) ) 1 Throughou he subsequen discussions, le c (i) c (i) { f (x + 1 i I ) ( f (x + ) ), (17) (i). (18) c(i) Using hese noaions, we have he following lemma. be given by. (19) 9

10 Lemma 1. Le {x be a sequence generaed by CL-FOBOS, and suppose ha Assumpions 1 and 2 hold. Then, he following inequaliy holds: R cl (T ) U γ (T ) i I ) 1 c (i) 1 ) i I ) +nks. Proof. To prove his lemma, we evaluae he firs erm of he righ-hand side in (17); ha is, {( f (x ) + ) ( ) f (x + i I 1 Firs, we obain ha ). (20) 1 1 T (i) 1 1 i I + + r i ( T (i) +1 r i ( T (i) +1, (21) where he las equaion follows from (18) and he propery ha r i (x is independen of. From he condiions (i), (13), and (18) of Assumpion 2, we can evaluae he second erm in (21) as r i ( T (i) +1 K T ri ( nks. Moreover, because c (i) 0 for i / I, we obain ha c (i) ) c (i) ). (22) i I 10

11 By subsiuing (21) and (22) ino (20), we have ha {( f (x ) + ) ( ) f (x + i I 1 1 ) {( f (x ) + ) ( ) f (x + ) i I i I {( f (x ) c (i) ) + U γ (T ) + 1 ( i I ) ( c (i) ) f (x + 1 c (i) c (i) ) c (i) + 1 i I r i ( T (i) +1 1 i I +nks ) ) ( ) ) +nks, (23) where he las inequaliy of (23) follows from (16). Finally, by replacing c (i) from (19), we obain in (23) wih c (i) {( f (x ) + ) ( ) f (x + i I 1 U γ (T ) + 1 i I 1 c (i) ) ) +nks. From Lemma 1, we can now obain he upper bound of he regre R cl (T ) as where P (T ) and Q(T ) are defined by R cl (T ) U γ (T ) + P (T ) + Q(T ) + nks, (24) P (T ) Q(T ) 1 i I 1 1 c (i) ) ). (25) 1 i I ), (26) respecively. We will now evaluae he upper bounds of P (T ) and Q(T ). For his purpose, le us firs derive an upper bound for he erm 1 c (i). Lemma 2. Suppose ha Assumpion 2 holds. Then, we have ha 1 c (i) δ() 11

12 for i I, where δ() is given by δ() { 1 if K 1 K 1 2( K+1) oerwise. Proof. From he definiions of c (i) and c (i), we have for each i I, 1 c (i) 1 c(i) 1 j (i) α +1 α j. (27) Noe ha α j α holds for j, because i follows from Assumpion 2 (iv) ha α α 0. Then, i follows from (18) ha α j (i) +1 j α α j (i) +1 α ( (i) )α α 1. (28) Moreover, from (27) and (28), we obain ha 1 j (i) α +1 α j α j (i) +1 j α j (i) 1 +1(α j α ) α. (29) Nex, le us consider he upper bound of α j in (29). When > K 1, we have ha 0 < K + 1 (i) + 1, from Assumpion 2 (i). Then, we have ha α j α K+1 holds for 12

13 j (i) + 1,...,, which follows from Assumpion 2 (iv). Hence, (29) can be rewrien as 1 j (i) α +1 α j j (i) +1(α j α ) α j (i) +1(α K+1 α ) α k(i) (α K+1 α ) α (α K+1 α ) α ( K+1 α 0 α ) 0 α 0 K + 1 K + 1 K 1 K + 1( + K + 1) K 1 K + 1(2 K + 1) K 1 2( K + 1). (30) On he oher hand, when K 1, α j α 0 holds for any j. Then, i hen follows from (18) ha (29) can be evaluaed as α j (i) +1 j j (i) +1(α j α ) 1 α Consequenly, we have from (27), (30), and (31) ha 1 c (i) α j (i) +1(α 0 α ) α (α 0 α ) α 1. (31) δ(). Using Lemma 2, we can give an upper bound for P (T ) as follows. Lemma 3. Suppose ha Assumpions 1 and 2 hold. Then, he following inequaliy holds: P (T ) K(K K K + 1)R + K(K 1)R( ) 1 + log(t K + 1). 2 13

14 Proof. By applying Lemma 2 o he definiion (25) of P (T ), we ge ha P (T ) 1 K i I 1 i I 1 c (i) ) 1 c (i) ) K δ() ) 1 i I K δ() ) 1 K 1 { K ( 1) 1 1 ) +K { K 1 ), 2( K + 1) where he firs inequaliy follows from Assumpion 2 (i). Moreover, by using (12) we have ha K 1 { P (T ) K ( 1) ) r i (x (i) { K 1 ) +K ) 2( K + 1) ( 1) + K 1 KR 1 K(K 1)R 2 K K K The firs erm of he righ-hand side in (32) can be evaluaed as K 1 ( 1) 1 K 1 1 K + 1. (32) ( 1)d < K K K + 1. Moreover, he second erm of he righ-hand side in (32) is bounded above, as K T K+1 1 K i 1 + log(t K + 1). By subsiuing hese wo inequaliies ino (32), we obain he desired inequaliy. Finally, we now derive he upper bound of Q(T ) in (24). Lemma 4. Le {x be a sequence generaed by CL-FOBOS, and suppose ha Assumpions 1 and 2 hold. Then, i holds ha Q(T ) nks. Proof. To simplify, le Q (i) (T ) and Q (i) (T ) be given by Q (i) (T ) Q (i) (T ) T (i) 1 T (i) ) r T (i) +1 1 i ( ), κ (i) ), 14

15 respecively, where T (i) denoes he mos recen ieraion where was updaed prior o he T h ieraion, and each κ (i) is given by Firs, we firs show ha κ (i) { if i I 0 oherwise. { Q (i) (T ) + Q (i) (T ) Q(T ). (33) From he riangle inequaliy, we have ha { Q (i) (T ) + Q (i) (T ) T { (i) I follows from he definiion of κ (i) 1 T (i) 1 1 { Q (i) (T ) + Q (i) (T ) T (i) ) ) ) and (18) ha κ (i) T (i) T (i) 1 ) ) ) ) ) + κ (i) ) + κ (i) ) T (i) We see ha he las erm is Q(T ), and hence we have ha n 1. i I r T (i) +1 κ (i) i ( ) r i ( ) T (i) +1 ) κ (i) ) κ (i) ) ). { Q (i) (T ) + Q (i) (T ) Q(T ). From his inequaliy, we may evaluae boh Q (i) (T ) and Q (i) (T ) o obain he upper bound of Q(T ). Firs, we firs define some noaions o be employed in hese evaluaions. Le l(i, m) be he ieraion a which has been updaed m imes wih l(i, 0) 0. Furhermore, le M (i) T denoe he number imes ha has been updaed by he T h ieraion. Noe ha M (i) T saisfies l(i, M (i) T ) T (i), (34) 15

16 because he mos recen ieraion a which has been updaed before he T h ieraion is he same as ha a which has been updaed M (i) T imes. We consider he properies of he sequence { wih respec o l(i, m) and M (i) T. Because {x(i) is he sequence generaed by (10), is only updaed when i I. Therefore we have ha i I { l(i, m) (i) m 1,..., M T, (35) i / I { l(i, m) + 1,..., l(i, m + 1) 1 (i) m 1,..., M T. (36) In paricular, we have ha follows from (36) ha +1 x(i) holds a each ieraion such ha i / I. Then, i l(i,m+1) ( l(i, m) + 1,..., l(i, m + 1)). (37) Now, using hese noaions we evaluae will Q (i) (T ) and Q (i) (T ). Firs, we can use (35) o rewrie Q (i) (T ) as Q (i) (T ) T (i) 1 M (i) T 1 m0 T (i) ) l(i,m+1) 1 l(i,m)+1 i I ) ) M (i) T 1 m0 l(i,m+1) ri ( l(i,m+1) ). Here, noe ha l(i,m+1) ri ( l(i,m+1) ) l(i,m+1) l(i,m)+1 ri ( l(i,m+1) ), because k(i) l(i,m+1) l(i, m + 1) l(i, m). Therefore, we have ha Q (i) (T ) M (i) T 1 m0 l(i,m+1) l(i,m)+1 ) M (i) T 1 m0 l(i,m+1) l(i,m)+1 l(i,m+1) ) 0, (38) where he las equaion follows from (37). On he oher hand, from condiions (i) and (13) of Assumpion 2 we have ha Q (i) (T ) r T (i) +1 T K+1 i ( ) ) KS. (39) Furhermore, from (33), (38), and (39), we obain ha Q(T ) nks. Proof of Theorem 1. I is obvious ha Theorem 1 follows from Lemma 3, Lemma 4, (16), and (24). 16

17 Theorem 1 yields ha lim T R cl (T ) T 0. We can regard CL-FOBOS as no only an online gradien mehod, bu also as a sochasic gradien mehod for he following sochasic programming problem: min E ( θ(x, z) ), (40) where z is a random variable, and θ is he loss funcion wih respec o z and a parameer x. Problem (40) is approximaely equivalen o he following problem: min ϕ(x) 1 T θ(x, z ), (41) where z represens a random variable for each sample. Now, we obain he following heorem in a way similar o [15]. Theorem 2. Le {x be he sequence generaed by CL-FOBOS for problem (41), and suppose ha Assumpions 1 and 2 hold. Then, we have ha [ ( 1 lim E ϕ T T 1 x )] ϕ 0, 1 where ϕ is he opimal value of problem (40). 5 Numerical Experimens In his secion, we presen some numerical resuls using CL-FOBOS in order o verify is efficiency. The experimens are carried ou on a machine wih a 1.7 GHz Inel Core i7 CPU and 8 GB memory, and we implemen all codes in C++. We se he iniial poin x 1 o 0 in all experimens. We compare he performance of CL-FOBOS wih L-FOBOS [10] for binary classificaion problems. We use he logisic funcion f (x) log ( 1 + exp( b a, x ) ) and he L1 regularizaion erm r(x) λ n x(i), where λ is a posiive consan ha conrols he sparsiy of soluions, and (a, b ) denoes he h sample daa. We use an Amazon review daase, called he Muli-Domain Senimen Daase [3], for he experimens. This daase conains many merchandise reviews from Amazon.com. We aemp o divide he reviews ino wo classes, hose expressing posiive impressions of he merchandises and hose ha are negaive. The daa for he h review is composed of a vecor a R n and a label b {+1, 1. The vecor a R n is consruced by he so-called Bag-of-Words process, and a (i) represens he number of imes ha he word represened by i appears in he h senence of he review. Hence, a is regarded as a feaure vecor of he h senence of he review. Meanwhile, he label b indicaes he evaluaion of he merchandise given by he auhor of he h review, where + 1 represens a posiive impression and 1 represens a negaive one. In he experimens, we only use he Book domain from he daase. The book domain is comprised of 4,465 samples, and he oal number of words in all samples is 332,440. Each feaure vecor a ( 1,..., M) conains a leas 11 (a mos 5,412) nonzero elemens, and he average number of nonzero elemens is

18 The number of samples, 4,465, seems small for comparing online gradien mehods. Therefore, we consruced a large daase using he 4, 465 samples. Tha is, we randomly choose 100, 000 samples from he original 4,465, and we employed his as he daase for he experimens. We compare CL-FOBOS wih L-FOBOS in he following four aspecs: he average of he oal loss defined in (42), he rae of correc esimaions, he number of nonzero componens in {x, and he run ime. T 1{f (x ) + n R(T r(i) (x i ) ). (42) T Noe ha R(T ) represens a kind of he regre R(T ). The rae of correc esimaions is defined as follows. We esimae he class of a sample a each ieraion using a, x, and hen coun he number of correc esimaions; ha is, he number of samples such ha b a, x > 0. The rae of correc esimaions is hen he number of correc esimaions divided by he number of samples ha have been observed. We esed boh mehods wih α α 0, where α 0 denoes he iniial sepsize, and is se beween and 1.0 wih widh of We presen he resuls achieved by each mehod wih he bes choice of α 0 ; ha is, we choose α 0 such ha smalles R(T ) is obained. Tables 1-6 show he run ime, R(T ), and he rae of correc esimaions for boh mehods. All of he numbers in he ables indicae an average aken over 10 rials for 10 randomly generaed daases. We can see from Tables 1-6 ha he run ime of L-FOBOS is shorened as K becomes larger. This is because for large K L-FOBOS rarely carries ou Sep 2, which requires O(n) compuaions. We also observe ha he rae of correc esimaions for CL-FOBOS is equal o or beer han ha of L-FOBOS when λ This shows ha CL-FOBOS effecively explois he informaion given by r i when λ is sufficienly large. However, CL-FOBOS achieved a remarkably low accuracy rae when λ is large. The main reason for his is ha mos of he values become close zero a each ieraion in CL-FOBOS, whereas L-FOBOS only considers he loss funcions oher han a ieraions such ha mod K 0. Thus, he resul does no imply ha CL-FOBOS is inferior o L-FOBOS. Nex, we consider he relaion beween R(T ) and he run ime. Figure 1 presens a summary of Tables 1-6 in erms of R(T ) and he run ime. From his figure, we can confirm he radeoff of L-FOBOS as we change K when λ Furhermore, here exiss no parameer K ha enables L-FOBOS o ouperform CL-FOBOS in erms of boh he run ime and R(T ). These resuls indicae ha, o a cerain degree, CL-FOBOS is more efficien when λ is large. In addiion, CL-FOBOS does no require he uning up of he parameer K, which is an advanage of he proposed mehod. On he oher hand, when λ is smaller han , such a radeoff does no occur, because he influence of he L1 regularizaion erm is very small. Finally, we noe he sparsiy of soluions {x generaed by boh mehods. Figure 2 illusraes he percenage of nonzero componens in {x when λ and λ The red lines indicae he performance of CL-FOBOS, and he green lines indicae ha of L-FOBOS. When λ , we employ K 400 for L-FOBOS wih an iniial sepsize of α Furher, when λ 10 5, we choose K 750 for his mehod, wih α These candidaes for K and α are chosen because L-FOBOS achieves he same run ime wih hem as CL-FOBOS. In his figure, we observe ha he green lines are jagged. This is because 18

19 Table 1: Resuls for each algorihm, λ 10 1 Algorihm Name α 0 Run Time R(T ) Rae of Righ Esimaions L-FOBOS (K 100) L-FOBOS (K 500) L-FOBOS (K 1000) CL-FOBOS Table 2: Resuls for each algorihm, λ 10 2 Algorihm Name α 0 Run Time R(T ) Rae of Righ Esimaions L-FOBOS (K 100) L-FOBOS (K 500) L-FOBOS (K 1000) CL-FOBOS Table 3: Resuls for each algorihm, λ Algorihm Name α 0 Run Time R(T ) Rae of Righ Esimaions L-FOBOS (K 100) L-FOBOS (K 500) L-FOBOS (K 1000) CL-FOBOS Table 4: Resuls for each algorihms, λ Algorihm Name α 0 Run Time R(T ) Rae of Righ Esimaions L-FOBOS (K 100) L-FOBOS (K 500) L-FOBOS (K 1000) CL-FOBOS Table 5: Resuls for each algorihm, λ 10 5 Algorihm Name α 0 Run Time R(T ) Rae of Righ Esimaions L-FOBOS (K 100) L-FOBOS (K 500) L-FOBOS (K 1000) CL-FOBOS

20 Table 6: Resuls for each algorihm, λ 10 7 Algorihm Name α 0 Run Time R(T ) Rae of Righ Esimaions L-FOBOS (K 100) L-FOBOS (K 500) L-FOBOS (K 1000) CL-FOBOS (a) λ 10 1 (b) λ 10 2 (c) λ (d) λ (e) λ 10 5 (f) λ 10 7 Figure 1: Relaion beween R(T ) and he run ime 20

21 (a) λ (b) λ 10 5 Figure 2: The number of nonzero componens in {x x becomes radically sparse when mod K 0, on accoun of he lazy evaluaion. In paricular, when λ he soluion x obained by L-FOBOS becomes approximaely zero a such ieraions. Meanwhile, when we se λ 10 5, we see ha boh mehods generae more dense soluions, because he L1 regularizaion is almos ignored. In addiion, we can observe he resuls of classificaion for he original 4, 465 samples using several soluions, such as x 399, x 400, x 9999, x 10000, x 99999, and x The rae and number of correc esimaions for each soluion are summarized in Table 7. Moreover, Figure 3 illusraes he resuls. Table 7: Rae of correc esimaions, λ The numbers in parenhesis denoe he number of correc esimaions. Soluion Number x 399 x 400 x 9999 x x x L-FOBOS (K 400) 68.7% (3,066) 57.3% (2,953) 78.4% (3,501) 77.9% (3,476) 79.2% (3,537) 79.0% (3,529) CL-FOBOS 67.5% (3,015) 67.4% (3,020) 78.5% (3,504) 78.5% (3,506) 79.1% (3,533) 79.0% (3,530) Figure 3: Number of correc esimaions We observe from Table 7 and Figure 3 ha he rae of correc esimaions decreases in L-FOBOS following is lazy evaluaion. This means ha L-FOBOS canno achieve he wo 21

22 aims of generaing sparse soluions and minimizing he oal loss simulaneously. On he oher hand, CL-FOBOS is well-balanced. Tha is, i always creaes sparse soluions, and is accuracy is improved as i observes samples. In conclusion, we confirm ha he proposed mehod, CL-FOBOS, is more efficien for problems ha are srongly influenced by he funcion r. 6 Conclusion In his paper, we have proposed a novel algorihm, CL-FOBOS, for large-scale problems wih sparse gradiens. Furhermore, we have derived a bound on he regre of CL-FOBOS. Finally, we confirmed he efficiency of he proposed mehod using numerical experimens. As a direcion for fuure work, i would be worh invesigaing a reasonable sepsize rule o be deermined before CL-FOBOS sars. References [1] Bersekas, P, D. : Incremenal Gradien, Subgradien, and Proximal Mehods for Convex Opimizaion: A Survey, Lab. for informaion and decision sysems repor LIDSP-2848, MIT, pp.1-38, [2] Bishop, C. : Paern Recogniion and Machine Learning, Springer-Verlag New York, [3] Blizer, J., Dredze, M., Pereira, F. : Domain Adapaion for Senimen Classificaion, In proceedings of he 45h annual meeing of he associaion of compuaional linguisics, pp , [4] Blondel, M., Seki, K., Uehara, K. : Block Coordinae Descen Algorihms for Large-scale Sparse Muliclass Classificaion, The journal of machine learning research volume 93, pp , [5] Boou, L. : Online Algorihms and Sochasic Approximaions, Online learning and neural neworks, pp. 9-42, [6] Cesa-Bianchi, N. : Analysis of Two Gradien-based Algorihms for On-line Regression, Journal of compuer and sysem sciences volume 59, pp , [7] Dredze, M., Crammer, K., Pereira. : Confidence-weighed Linear Classificaion, Proceedings of inernaional conference on machine learning, pp , [8] Duchi, J., Singer, Y. : Efficien Online and Bach Learning Using Forward Backward Spliing, The journal of machine learning research volume 10, pp , [9] Duchi, J., Shalev-Shwarz, S., Singer, Y., Tewari, A. : Composie Objecive Mirror Descen, Conference on learning heory (COLT), pp , [10] Langford, J., Li, L., Zhang, T. : Sparse Online Learning via Truncaed Gradien, The journal of machine learning research volume 10, pp ,

23 [11] Lipon, Z., Elkan, C. : Efficien Elasic Ne Regularizaion for Sparse Linear Models, arxiv: , 24 May [12] Togari, Y., Yamashia, N. : Regre Analysis of Online Opimizaion Algorihm ha performs Lazy Evaluaion, Bachelor hesis, Kyoo Universiy, March 2014, hp://www-opima.amp.i.kyoo-u.ac.jp/papers/bachelor/2014_bachelor_ ogari.pdf [publish in Japanese]. [13] Togari, Y., Yamashia, N. : Regre Analysis of Forward-Backward Spliing Mehod wih Componen-wise Lazy Evaluaion, In Proceedings of he Spring Meeing of he Operaions Research Sociey of Japan, 2015 [publish in Japanese]. [14] Shalev-Shwarz, S. : Online Learning and Online Convex Opimizaion, Foundaions and rends in machine learning volume 4, no 2, pp , [15] Xiao, L. : Dual Averaging Mehods for Regularized Sochasic Learning and Online Opimizaion, The journal of machine learning research volume 11, pp , [16] Zinkevich, M. : Online Convex Programming and Generalized Infiniesimal Gradien Ascen, Proceedings of he wenieh inernaional conference on machine learning (ICML) pp ,