Matrix regularization techniques for online multitask learning

Transcription

1 Matix egulaization technique fo online multitak leaning Alekh Agawal Compute Science Diviion UC Bekeley Pete L. Batlett Compute Science Diviion Depatment of Statitic UC Bekeley Alexande Rakhlin Compute Science Diviion UC Bekeley Abtact In thi pape we examine the poblem of pediction with expet advice in a etup whee the leane i peented with a equence of example coming fom diffeent tak. In ode fo the leane to be able to benefit fom pefoming multiple tak imultaneouly, we make aumption of tak elatedne by containing the compaato to ue a lee numbe of bet expet than the numbe of tak. We how how thi coepond natually to leaning unde pectal o tuctual matix containt, and popoe egulaization technique to enfoce the containt. The egulaization technique popoed hee ae inteeting in thei own ight and multitak leaning i jut one application fo the idea. A theoetical analyi of one uch egulaize i pefomed, and a eget bound that how benefit of thi etup i epoted. 1 Intoduction The poblem of multitak leaning i a cenaio whee the leane eceive example dawn fom moe than one tak. A algoithm fo ingle-tak poblem ae eadily available, the implet appoach i to olve each of the tak independently of the othe. Howeve, if tak ae elated, ignoing the common tuctue mean thowing out ueful infomation. Fom the algoithmic point of view, the hallmak of multitak leaning i developing way of exploiting tak elatedne. Fom the theoetical point of view, the goal i to quantify the impovement, taking a a baeline pefomance when tak ae leaned epaately. In thi pape, we povide a new algoithm and quantify the gain of leaning the tak togethe. We note that the algoithm we peent i inteeting in it own ight and can be employed in etting beyond multitak leaning. The multitak leaning poblem ha een a lot of wok in ecent yea (e.g. [4], [3], [2]). A common theme of all thee appoache i to model the notion of tak elatedne via an aumption about the low ank of the data matix. Howeve, thee appoache often eult in non-convex optimization poblem, which cannot be olved exactly in a computationally efficient manne. Alo, no ignificant theoetical analyi ha been done in thi etup. In the online etup, thi poblem wa looked at in [7], [1] and ecently in [5]. In online multitak leaning poblem with expet, the leane eceive a tak id and a vecto of the loe of each expet at evey time tep. The lo of the leane i the expected lo unde hi ditibution ove the expet fo that tak. The notion of tak elatedne i a little diffeent in thi etup, and will be decibed at length in the following ection. Coeponding autho. 1

2 The autho in [1] peent an expet algoithm imila to weighted majoity fo obtaining the optimal eget bound in thi poblem. Thi optimal algoithm howeve, involve an NP-Had computation, and i appoximated by an MCMC pocedue. Cavallanti et al [5] ued matix egulaization appoache that wee ued in the batch etup to obtain efficient algoithm, but thei eget bound cale much woe than the optimal, in the wot cae incuing a linea dependence on the numbe of tak. In thi pape, we will look at ome new matix egulaization idea that ae moe uited to thi leaning poblem. In paticula, we will decibe pectal and tuctual matix egulaization appoache, and how how the latte allow u to obtain efficient low eget algoithm. Summay of eult: In thi pape we obtain a computationally efficient algoithm whoe eget cale no woe than O( KmT log N) fo leaning K tak with N expet fo each, with paamete m quantifying tak imilaty. We will how that in ome inteeting egime of K, N value, thi come quite cloe to the optimal eget. Thi i a ignificant impovement ove the eult of Cavallanti et al [5] whoe algoithm yield an uppe bound on the eget caling a O((K m) T log min{k, N}). The latte bound can be linea in the numbe of tak in the wot cae, implying little gain ove the baeline of leaning tak independently. 2 Setup In thi ection, we intoduce ome notation to be ued thoughout the pape and povide a pecie tatement of the poblem. We denote vecto by lowe cae lette uch a u, w, x, l. Uppe cae lette typically denote matice. We will uually ue U, W to efe to matice ued by the compaato and playe epectively, and thee matice will be non-negative with ow adding up to 1 unle othewie pecified. W i will be ued to denote the i th column of matix W. The notation S m and S m intoduced below will be ued intechangibly fo the compaato cla pecified in tem of expet o the compaato matix U. Pai p, q and, will be ued to efe to conjugate exponent, i.e. 1/p + 1/q = 1 and 1/ + 1/ = Poblem Specification Let u now fomalize the game by decibing the notion of tak and elatedne. Since the only aumption that elate the equence in a claical online etup i the etiction of compaato to a ingle bet expet, it i natual to define a tak a a et of example that ue the ame bet compaato. Of coue, thi aumption i meaningle unle the leane know which example would be uing the ame compaato. Hence, we aume that the leane, along with the pediction of the expet, i alo told the identity of the tak fom which thee pediction wee geneated. In thi aticle, we define a etup whee thee ae N diffeent expet whoe pediction we eceive at each tep, fo one of K diffeent tak. Fo the cae of a ingle tak, the minimax optimal eget bound i T/2 ln N. Without any futhe aumption on tak, it i clea that K T/2 ln N i the bet eget that can be achieved. Howeve, we hope to do bette when the tak ae elated. One way to fomalize thi i by aying that the actual numbe of diffeent tak i jut m K. Thi amount to aying that the compaato i allowed to have jut m intead of K diffeent pedicto. Howeve, the leane till get a tak identity between 1,..., K. The aim of the leane i to dicove the tak imilaity o a to get a eget bound with ome dependence on m. To get an idea of the optimal dependence on m and othe paamete, we need to conide the effective compaato cla of ou poblem. 2.2 Compaato cla In the expet etting, the eget i typically defined a: ˆL t min i 2 L t i (1)

3 whee ˆL t i the lo of the leane at time t. In the multitak poblem, we wih to genealize thi to m diffeent compaato. Let [K] = {1,..., K} and S m := {S [K] : S = m}. Then clealy, any compaato fom the cla S m ue only m diffeent expet, and thu fom ou compaato cla. The eget of ou poblem i defined a: R T = ˆL t min K S S m k=1 min i S t T k L t i,k (2) whee T k = {t : k t = k} and k t i the tak id at time t. In the ingle tak etting, the eget bound contain the facto ln N, which i een to be the log cadinality of the compaato cla fo that poblem (a the optimal compaato pick jut one of the expet). Then it i natual to ak if we can compute the cadinality of thi compaato cla. The hope would be to obtain eget bound caling with the log of it ize again. It i not too had to ee that S m ( N m) m K. In fact, it i hown in [1] that thi etimate i aymptotically of the ight ode, and thu log S m = Θ ( m log N m + K log m). The autho futhe howed that the weighted majoity algoithm can be eaily adapted to intead lean ditibution ove the element of S m to indeed attain a eget baed on thi quantity. Howeve, the algoithm need to maintain ditibution ove an exponentially lage cla of expet now, and it wa hown that computing and updating thee weight i NP-Had. We can now etup the game of online multitak leaning a in Figue 1 1: fo t = 1 to T do 2: Aveay give tak id k t. 3: Playe pecifie it ditibution ove expet fo the tak ˆp t. 4: Pleaye incu the lo ˆp t l t fo the lo vecto l t geneated by the adveay. 5: end fo Figue 1: The online multitak leaning game A light genealization to the above etup will be ued in the late ection of thi pape. Conide the extended compaato cla S m := {S [K] : S m}. By a imila agument a above, the ize of thi cla bounded by N m m K. Thu log S m = Θ (m log N + K log m). 3 Matix Regulaization fo multitak leaning The main idea behind applying matix egulaization technique to the multitak leaning poblem tat with fit epeenting the compaato, pedicto and example a K N matice intead of vecto. A we ty to pecify a ditibution ove expet fo each tak, all the entie of the matix have to be non-negative, and each ow hould add up to 1. The matix fo the compaato can be een a a 0-1 matix with only one non-zeo enty pe ow which identifie the optimal expet fo each tak. Clealy at mot m column in thi matix can have non-zeo entie by the aumption of the peviou ection, if the compaato i in the cla S m. Thi i becaue evey non-zeo column of thi matix epeent an expet ued fo at leat one tak. So to make ue that the total numbe of expet ued aco all tak i mall, we have to keep the numbe of non-zeo column mall. The leane matix epeent the K vecto coeponding to it pedictive ditibution ove the expet fo the vaiou tak. Finally, fo homogeneity, we epeent the vecto of loe at each time a the K N matix X t, whee X t ha only one non-zeo ow coeponding to the tak fom which the example i dawn. Now conide the compaato matix. Thi matix ha at mot m non-zeo column. Auming m < N, K, the matix, thu, ha a ank of at mot m. When competing againt a compaato of low ank, it make ene fo the leane alo to etict itelf to only thoe et of ditibution ove tak which motly give ie to low ank matice. Thi i intuitively eaonable, a thi allow the leane to etict itelf to a malle ubcla of matice, and thu exploit the paticula tuctue of the compaato. A the eget of online algoithm 3

4 typically cale with the ize of the cla that they each ove, thee i hope that uch an algoithm looking ove a mall cla will alo achieve a lowe eget guaantee. In fact, if the leane maintain a pediction matix W of ank m, then the matix can be factoized a W = AB whee A i K m and B i m N. The matix A can be een a pecifying a mapping fom tak to thei tue tak id (of which thee ae only m), while B give a map fom thee to the bet expet fo each tue tak. Howeve, it i immediate that the low ank aumption allow moe geneal tuctue. In paticula, a tak could be a linea combination of two othe tak, and thi would till keep the matix low ank. Thi geneality i alo quite intuitive fom a point of view of a notion of tak-elatedne. Since, we ae doing online update, if we have an update uch that once initialized with a ank m matix, it will alway keep the matix ank malle than o equal to m, then we could exploit uch an update in thi poblem by initializing the leane at a ank m matix. Thi idea ha been exploited fo leaning low ank kenel matice in an online fahion in [8]. The key idea i to define uitable Begman divegence ove the pace of matice. Fo two quae ymmetic poitive emidefinite (pd) matice X, Y define the von Neumann and Bug divegence, epectively, a D V N (X, Y ) = T (X log X X log Y X + Y ) (3) D Bug (X, Y ) = T (XY 1 ) logdet(xy 1 ) n (4) Note that the Bug divegence i a natual genealization of log-baie to quae ymmetic matice, wheea the von Neumann divegence i the analogue of the entopy function. The inteeting fact i that if the matix Y ha a ank m, then any matix X with a finite Bug divegence (D Bug ) fom Y ha a ank m a well. Thi i een by witing the divegence a a function of eigenvalue. Fo a finite von Neumann divegence (D V N ) X need to have a ank m. Thi mean that if the online update ae done by minimizing the lo plu one of thee divegence, then the ank containt ae automatically enfoced. Unfotunately, thee divegence ae only defined fo quae ymmetic matice. An obviou way to extend thi to ectangula matice i by applying thee divegence to W W o (W W ) 1/2 which ae both quae ymmetic and pd. Unfotunately, thi lead to an algoithm fo which update ae had to analyze. In thi aticle, we will not exploe thi thought futhe, although it i conceivable that thi appoach might enjoy the a nea optimal eget bound. A emak about the tuctue of the compaato et i in ode at thi point. Note that the ank containt i a non-convex containt; the ank of a um of two matice can be a lage a the um of thei ank. In fact, it i not had to contuct full ank matice uing matice fom S m. So if we naively ty to maintain a low ank by eticting ou online optimization to thi et, we could be in touble. While the divegence given above povide a way of optimization unde ank containt in the cae of quae matice, we cannot in geneal extend any algoithm to wok within thi ubet by focing it to optimize ove jut S m. A natual tategy that i often ued in uch poblem i to intead augment the objective function with a egulaize which take mall value on the egion of choice matice in S m in thi cae. While pefoming egulaized optimization with uch function doen t guaantee that the playe will tay in the et, it foce a pefeence fo taying in the et. Below we will ee ome matix nom that ty to fulfil thi intuition. While it i poible to get to abitay ditibution ove expet by pefoming optimization unde thee egulaize, we will ee that the pefeence they model to tay in the et i tong enough and uffice to obtain non-tivial eget guaantee. 3.1 Stuctual and pectal matix nom In the peviou ection, while decibing the containt we want to impoe on the leane and compaato, we went back and foth between the idea that the matix decibing thee ditibution can be ank m o can have at mot m non-zeo column. While the fome agument lead to the idea of egulaizing the eigenvalue o that a mall numbe of them ae non-zeo, the latte encouage a moe diect egulaization of the entie of the matix in uch a way that the numbe of non-zeo column in the matix i mall. 4

5 We efe to nom (o geneal egulaizing function) acting on the eigenvalue a pectal nom, a they only depend on the pectum of the matix. The nom (egulaize) that act diectly on the entie of the matix to enfoce a pecific tuctue, uch a a mall numbe of non-zeo column, ae efeed to a tuctual. While pectal nom ae extenively tudied in liteatue, ou undetanding of tuctual nom i elatively nacent, and in the following ection we will put foth one candidate popoal that uit thi poblem. Clealy in thi etup, containing the pectal nom to be mall allow all the matice that would be allowed by the analogou tuctual containt, and moe. Thu it eem natual to hope that by moving to the moe diect tuctual egulaization, we might be able to obtain bette eult. A we will ee in the following ection and the eget analyi, thi intuition i tue indeed. 3.2 Rank egulaizing matix nom The containt of low ank in the matix domain i vey imila to the notion of paity in the cae of vecto. Fo vecto, it i well-known that minimizing the l 1 nom lead to pae olution. Futhemoe, a dicued in Chapte 11 of [6], egulaization with an l p nom with p uitably cloe to 1 lead to the optimal eget bound up to contant facto in the vectoial etup. It i natual, theefoe, to ak if one can define appopiate nom on matice that egulaize it ank to give low eget algoithm fo online multitak leaning. We will dicu two uch nom below and define an algoithm and pove a eget bound fo the latte. The geneal cheme of online leaning algoithm that we will be conideing ae algoithm of the fom: W t+1 = agmin W K N D(W, W t ) + ηt (W T X t ) (5) whee D i the Begman divegence induced by one of the nom to be decibed below. K N i the et of all K N matice which have all entie non-negative, and element of each ow add upto 1, i.e. fom a ditibution. Obeve again that the pojection ae onto the pace of all ditibution ove expet and not ove a eticted ubet due to the non-convexity of the ank containt a explained above. Alo note that we ae uing a linea lo hee, which doe not educe the geneality a it i well undetood that cuved lo function only help the leane athe than the adveay in thi etup. 3.3 Schatten L p penom Note that a ank m matix ha exactly m non-zeo ingula value in it ingula value decompoition (SVD). Hence, doing an l p egulaization on the vecto of ingula value with p cloe to 1 can be hoped to enfoce only eveal non-zeo ingula value, leading to low ank matice. Fomally, define: ( W 2 S p = σ i p ) 2/p = T ((W W ) p/2 ) 2/p (6) whee = min{k, N} and σ i ae the ingula value of W. The econd equality follow fom the well-known fact that the eigenvalue of W W ae the quaed ingula value of W. Thi i the nom ued by Agyiou et al[4], howeve in the tochatic etup, and no theoetical analyi i povided. Recently Cavallanti et al[5] caied out an analyi with thee nom fo a cloely elated multi-view poblem. It i conceivable that a imila analyi extend to the multitak poblem too, but i not dicued in thei pape. It tun out howeve, that thee i anothe nom much moe conducive to analyi fo thi poblem, and uited bette to ou poblem intuition a agued below, which will be the main object of tudy in thi pape. 3.4 Matix (, p) nom fo tuctual egulaization Fo a K N matix W, let W i and W j efe to the i th column and j th ow ep. of the matix. The (, p) nom of a matix i given a: 5

6 ( N W 2,p = 2/p W i ) p (7) It i eay to how that the above definition i indeed a nom on the pace of matice. Lemma 1.,p i a nom on the pace of matice fo, p 1. Poof. It i clea fom the definition that W,p 0 and i 0 iff W 0. Thu we only need to veify the tiangle inequality. Uing the tiangle inequality on the nom, W + U,p = W j + U j p 1/p ( W j + ) U j p Thi tem can now be een a the p nom of a um of two vecto of length N each with the j th enty of one vecto being W j and the econd being U j. Then by the tiangle inequality of p nom on thee two vecto we get W + U,p W j p 1/p = W,p + U,p 1/p + U j p which give u the tiangle inequality, Thu,p i indeed a nom on the pace of matice. It hould be noted that thi nom genealize the (2,1) nom of [3]. The fit thing to note i that except fo pecific choice of, p, thi i not a pectal nom in geneal. Thi nom diectly act on the entie of the matix and can be diffeent fo two diffeent matice with the ame eigenvalue. Howeve, it doe enfoce the ight tuctual popetie on the matix a explained below. The, p nom i a natual genealization of the l p nom to matice, whee we ue nom on column, and then take a p nom of thee value. To ee why thi i intuitive, conide the cae of 0-1 matice, with = and p = 1. Then the nom of a column i 1 if it ha at leat 1 non-zeo enty, 0 othewie. Taking an l 1 nom of thee value coepond to counting the numbe of expet that ae being ued. While competing with compaato in S m, thi i exactly the quantity we want to keep below m. Hence thi nom doe eem to captue the ight intuition in ou poblem. That thi i not a pectal nom might make it look le attactive on the fit glance, but make it much moe amenable to analyi. It might thu eem that uing the afoementioned value of and p would lead to the optimal eget bound. Howeve, both thee value ae not uitable fo analyi. Indeed, the l 1 nom i not tictly convex, and, futhemoe, ou analyi equie, p 2. Thu we leave the choice of thee exponent open fo now, and hope to tune them to obtain the optimal bound once the analyi i complete. An impotant popety of, p nom i alo that fo caefully tuned value of, p they give a malle nom to the matice in S m than the matice outide. Thi i cucial to ou algoithm. Note that in Equation 5, we only poject on the pace of ditibution matice K N. Thi mean ou leane can in geneal be outide S m, which eentially mean that the poblem tuctue i not being adequately exploited, and we cannot hope fo a ignificantly lowe eget than olving the tak indpendently in the wot cae. Howeve, if matice in the et S m have a mall value of the nom, then it i eay to how that the pojection ha a much geate chance of landing inide thi et than outide. Thi popety of the nom play a cucial ole in enuing a low eget of thi leaning pocedue.. 1/p 6

7 4 Reget analyi Let L T (U) fo a matix U denote T T (U T X t ), the cumulative lo uing U fo pediction. Alo we ue L T = T T (W t T X t ) to be the cumulative lo of ou algoithm. Fo any matix U, eget R T (U) with epect to U i defined by L T L T (U). We will begin by tating the main eult of thi aticle, which will then be poved uing a eie of malle lemma. Theoem 1. Conide the leane uing (5) with the D the Begman divegence defined with epect to the (, p) nom. Let L T be it cumulative lo afte T tep. Suppoe that thee ae N expet and K tak. Let 1 < p < 2, and the lo of any expet at any time tep be bounded by κ. Then fo all U S m, all T and all η > 0: L T L T (U) + 1 ( 2η m2/p 2/ K 2/ + κt η 1 + p ) p 1 2 N 2(p 1)/p (8) The fit thing we need to obtain in ode to ue thi nom in the algoithm of (5) i to deive the Begman divegence induced by thi nom. It uffice to find the dual nom fo thi pupoe. It tun out that the dual of (, p) nom i the (, q) nom whee and q ae the exponent dual to and p epectively. We begin by poving a Hölde inequality fo thi nom. Lemma 2. Conide two matice A and B, each of ize K N. Then we have: whee and q ae conjugate to and p, epectively. T (A B) A,p B,q (9) Poof. The eult follow fom a imple application of Hölde inequality fo vecto. T (A B) = A j T B j A j T B j N A j B j (10) (Uing Hölde inequality fo vecto on each element of the um) 1/p A j p 1/q B j q (Hölde inequality on vecto of nom) = A,p B,q (11) We can now imply deive the dual a indicated ealie. Lemma 3. Let F (A) = 1/2 A 2,p. Then it Legende-Fenchel dual i given by F (B) = 1/2 B 2,q Poof. The dual function i defined a: F (B) = up A up A = up A { T (A T B) 1 } 2 A 2,p { A,p B,q 1 } 2 A 2,p { 1 2 B 2,q 1 2 ( B,q A,p ) 2 } (12) (13) 7

8 whee the inequality follow fom Lemma 2. Conide a paticula choice of A in (13): A = B ( 1) B j (q ) B (2 q),q. (14) We claim that thi choice a) achieve the upemum in (13) and b) tun the inequality leading to (12) into an equality. If both of thee point ae veified, the tatement of thi lemma would follow. Let u tat by howing a). A,p = A j p ( K ( = Uing conjugacy of p, q and,, we ee that p = ( K ( A,p = = B 2 q,q = B 2 q,q = B 2 q,q B 1 1/p B 1 B j (q ) B (2 q),q ) ) p/ 1/p q q 1 and = 1. Subtituting thi above, we obtain B j (q ) B j (q )q/(q 1) B j (q )q/(q 1) B j q = B 2 q,q B q 1,q = B,q. (q 1)/q ) B (2 q) 1,q ( K B ) q( 1)/(q 1) ) q( 1)/(q 1) B j q( 1)/(q 1) (q 1)/q (q 1)/q (q 1)/q Hence, if A i defined a in (14), the non-negative econd tem of (13) vanihe, yielding F (B) 1 2 B 2,q and veifying a). 8

9 Obeve that b) amount to howing tightne of two inequalitie in Lemma 2. Conide the quantity A j p B = j ( 1) B j (q ) B (2 q),q = B p(2 q) (,q B j p(q ) K = B p(2 q) (,q B j p(q ) K = B p(2 q),q = B,q p(2 q) B j p(q ) p B ( 1) B B j p( 1) B j p(q 1) = B p(2 q),q B j q = B j q B p,q B q(1 q)/(q 1),q B j q = B q A p,p,q Thi mean that Aj p A = Bj q p,p B q,q Futhemoe, uing an intemediate point ( ), ) p/ ) p( 1)/ which make the application of Hölde inequality in (11) tight. A A j = ( B ( 1) = B( 1) = B B j B j (q ) A j B j (q ) B j (q 1) ( ) ) B (2 q),q B (2 q),q B (2 q),q which make the inequality in (10) tight a well. Thu thi choice of A yield the deied eult, that i F (B) = 1 2 B 2,q completing the poof. The above poof give u the value of the matix A at which the maximum in the expeion fo the dual i attained. It i wothwhile to pend a minute inpecting the mapping between pimal and dual pace obtained above. It will be hown that the value of q, that yield a good eget have q. Fo uch value of q, the mapping fom dual pace to pimal pace tie to concentate mot of the ma of a ow in the column having laget entie in the dual matix B. Thu in mapping back to the pace of weight matice, ou nom implicitly tie to minimize the numbe of non-zeo column which i alo the numbe of expet ued. Thi futhe uppot ou intuition that thi nom i well-uited fo the poblem. Note that it i well known that if A attain the upemum in the definition of F (B), then A and B fom a pimal-dual pai, with A = F (B) and B = F (A). So, in paticula, we now have the deivative of ou 1 nom, A 2 A 2,p = A ( 1) A j (p ) A (2 p),p. We can now define a Begman divegence uing thi nom a the Begman function. We have D,p (W t 1, W t ) = 1 2 W t 1 2,p 1 2 W t 2,p T (V t T (W t 1 W t )), (15) whee V t i the dual image of W t a defined above, and D,p i the Begman divegence uing 2,p a the Begman function. A the lat peliminay eult, we deduce an uppe bound on the (, p)-nom that will be ueful in the late analyi. 9

10 Lemma 4. Let 1 < p < 2. Then U S m, U,p K 1/ m 1/p 1/. Poof. The fact that U S m implie that it ha at mot m non-zeo column. Alo, we have to pick at leat one expet fo each tak. Fit note that it uffice to look at jut 0-1 compaato matice, a the adveay will alway pick the et of m expet and tak aignment to thoe expet that eult in the mallet oveall lo ove the entie equence. With that in mind, we can et up the following optimization poblem: ( ) 1/p max n p/ 1 + n p/ n p/ m n 1,...,n m 0.t. n n m = K It i eay to how uing a quick econd deivative computation that thi objective i a concave function of the n i. So, we can compute the Lagange function, and et it deivative to zeo, which give u: ( ) (1 p)/p 1 np/ 1 i n p/ i = λ i whee λ i the Lagange multiplie, fo all i = 1... m. Thi mean that all the n i ae equal to K/m. Evaluating the nom uing a matix U with m non-zeo column, each having K/m one yield the deied eult. We ae now in a poition to pove the theoem. Poof of Theoem 1. Fo pedicto of the fom (5), the eget can be bounded a: R T (U) 1 η D,p(U, W 0 ) + 1 η D,p (W t 1, W t ) (16) Thi fom of eget bound i well-known, fo example Lemma 10 in the unpublihed lectue note [9] a well a [6]. Thu the key tep of the theoem i to bound the two divegence tem D,p (U, W 0 ) and D,p (W t 1, W t ). Note that we can take W 0 to be unifom (i.e. 1/N ove expet fo each tak). Then the entie of V 0 ( 1) ae unifom too, i.e., V 0 = W 0 W j 0 fo all column fo ome contant c. So we have (p ) W 0 (2 p),p = c fo all i, j a the column nom W j 0 ae ame T (V 0 T (U W 0 )) = i,j V 0 (U W 0 ) = c i,j ( U W 0 ) = 0 (a each ow of both U and W 0 add up to 1) So the linea tem of the fit Begman divegence tem in (16) i zeo, and hence thi divegence i lage when the compaato matix U ha a lage (, p) nom. Uing Lemma 4 thi happen peciely when the matix U ha exactly m non-zeo column, each with K/m one. Note that thi follow fom the aumption that each ow of the matix U fom a ditibution and hence um to 1. In thi cae, the nom of U i m 1/p 1/ K 1/. Dopping the negative W 0,p tem, the fit divegence tem i uppe bounded a fo all U S m. D,p (U, W 0 ) 1 2 U 2,p 1 2 m2/p 2/ K 2/ 10

11 We next tun to the divegence between iteate, D,p (W t 1, W t ). Fo convenience, define W t+1 a follow: W t+1 = agmin W D,p (W, W t ) + ηt (W T X t ) (17) i.e. the uncontained minimize of the optimization poblem. Then it i eaily hown that W t+1 = Π,p (W t ; K N ), whee Π,p (W ; S) i the Begman pojection of a matix W onto the et S, uing the Begman function 1 2 2,p. Alo, uing the Pythagoean inequality fo Begman divegence (ee fo example [6] Lemma 11.3), we can wite: D,p (W t 1, W t ) D,p (W t 1, W t ) + D,p (W t, W t ) Thi mean that we can bound the eget futhe fom (16) a: R T (U) 1 η D,p(U, W 0 ) + 1 η D,p (W t 1, W t ) (18) D,p (W t 1, W t ) (19) Now, if Ṽt i the dual vaiable coeponding to W t, we can eaily ague that Ṽt = V t 1 + ηx t. Thi can be een by diffeentiating the objective in (17) and etting it to zeo. It i a popety of Begman divegence (Pop of [6]) that D f (W t 1, W t ) = D f (Ṽt, V t 1 ) whee f i the convex conjugate of f and V t i the conjugate dual of W t, pecified by the gadient mapping V t = f(w t ). In the paticula context of thi poblem, thi popety i elevant a the update equation fom (5) ae vey imple in the dual pace. The dual update can be witten a Ṽt = V t 1 + ηx t 1 which i imply the gadient condition at optimality fom the fact that the deivative of D f (W, W t ) wt W i imply f(w ) f(w t ). Uing Lemma 5 that we will pove below, thee divegence ae bounded a: D,q (Ṽt, V t 1 ) η 2 ( + q 2) X t 1 2,q (20) Auming that the loe ae componentwie bounded at each time, and noting that we have non-zeo lo fo exactly one tak, we can bound X t 2,q with κn 2/q, whee κ i a bound on each enty of X t unifomly aco t = 1... T. Summing tem ove time, and uing the fact that = 1 and q = p p 1 complete the poof. Lemma 5. Let W t, W t+1 be a in (17), with divegence induced by the (, p) nom,with 1, p 2. Let V t and Ṽt+1 be the coeponding dual image, and, q be the dual exponent to, p ep. Then we have: D,q (Ṽt+1, V t ) η 2 ( + q 2) X t 2,q (21) Poof. The key idea a in mot poof of thi kind i to ue the fact that Begman divegence meaue the diffeence between a function and it fit ode Taylo appoximation. We know that thi diffeence i equal to the econd ode Taylo tem at ome intemediate point by the mean value theoem. So if we can unifomly bound the Heian matix of ou egulaize, it uffice to demontate a bound on the divegence. Let F (V ) = 1 2 V 2 F (V ),q. The Heian matix fo thi function i given by H (,) = V V. We can think of the Heian a eithe a 4-dimenional matix, o, a a 2-dimenional one. The latte i obtained by letting the index tand fo K (i 1) + j. With thi notation, the econd ode tem in the Mean Value Theoem i 1 2 (vec(v t Ṽt+1)) H( V )(vec(v t Ṽt+1)). (22) Hee vec i an opeato that tetche out it matix agument to a vecto and V = αv t + (1 α)ṽt+1 fo ome α [0, 1]. Witten a a ummation, thi i imply i,j,k,l (V t Ṽt+1) H( V ), (V t Ṽt+1). We now ue the fact that Ṽt+1 = V t + ηx t to wite thi a η 2 i,j,k,l (X t) H( V ), (X t ). 11

12 We now dop the ubcipt t and the ba fom V to eae the notation a bit. Let u look at a paticula enty of the matix H. By applying the chain ule, H, = F (V ) V V = ( V F (V )) = = V V (2 q),q V (2 q),q V + V l (q ) V + V ( 1) V Conideing each of the thee tem above, we get H, = (2 q) q V (2 2q) q,q V l (q ) V V l (q ) V ( 1) V ( 1) V (2 q),q V ( 1) V (2 q),q V l (q ) V j (q ) V ( 1) V l (q ) V ( 1) + I(j = l) V (2 q) (q ),q V l (q 2) V ( 1) il V ( 1) + I(i = k, j = l) V (2 q),q V l (q ) ( 1)V ( 2) = (2 q) V,q (2 2q) V j (q ) V l (q ) V ( 1) V ( 1) }{{} A + I(j = l)(q 1) V (2 q),q V j (q 2) V j 2(1 ) V ( 1) V ( 1) kj }{{} + I(j = l)(1 ) V (2 q),q V j (q 1) V j (1 2) V ( 1) V ( 1) kj }{{} + I(j = l, i = k)( 1) V (2 q),q V j (q ) V ( 2). }{{} D Now let u conide the ummation i,j,k,l H,X X by looking at the contibution of each of the tem A, B, C, D epaately. Fo the tem A, we ee that, X X (2 q) V (2 2q),q = (2 q) V (2 2q),q 0, a q 2 by the aumption that p 2. B C V j (q ) V j (q ) K V l (q ) V ( 1) X V ( 1) V ( 1) 2 12

13 A imila agument how that the contibution of tem C i negative a well, and hence thee two tem can be ignoed fo pupoe of getting uppe bound. So in ode to how an uppe bound, we jut need to account fo the contibution of the tem B and D. Conide the contibution of the tem B. Excluding the leading η 2, the um ove thee tem can be witten a: ( 1 V (q 2),q = 1 (q 1) V j q 2 V (q 2),q (q 1) V j q 2 1 ) 2 K ( K V )( 1)/ ( K X V ( 1) ( K V )( 1)/ (q 2)/q (q 1) n V (q 2) V j q,q Uing Hölde inequality with exponent ( K ) (q 1) V ( 1)/( 1) ( K ( K V )( 1)/ Uing Hölde inequality with exponent = (q 1) X 2,q i,k=1 ) 2 ( K X V ( 1) ( K V )( 1)/ q q 2 and q 2 X 1 and ) 1/ q X X kj V ( 1) V ( 1) kj ) 2q/2 Now we look at the econd tem. Uing imila application of Hölde inequality with lightly diffeent exponent, we can bound the tem whee i = k, j = l by ( 1) X 2,q. The othe two cae have been hown to be negative and ae thu dopped fom the uppe bound. Adding all the tem give u the deied uppe bound. Note that when q >, then the decompoition of the middle tem into tem B and C in the above poof i not needed, and we can get a lightly bette eget bound that involve jut (q 1) in place of ( + q 2). Howeve, thi doen t caue any ignificant change in the bound unle i vey lage, and hence we ue the lightly looe but moe geneal fom above fo futhe dicuion. 4.1 Optimal etting of paamete A the eade would have obeved, the application of Hölde inequalitie in the peviou tep citically elied on the fact that 2, q 2. Thu we need to have 1 <, p 2. Fom the bound of (8), it i clea that the bet etting of η i to balance the two tem. Thi allow u to ewite the bound a: R T (U) m 1/p 1/ K 1/ N (p 1)/p κt 2/q ( ) p 1 It i not obviou what the optimal etting fo, p i in geneal. Howeve, we can invetigate cetain egime in which it i poible ( to et the value of, p in a way that bing ou eget bound vey cloe to T ) the optimal eget of O (K log m + m log N). Conide etting K = N α fo ome α > 0. Then the optimal eget bound i dominated by the tem O ( T log mn α/2). If we plug thi value of K in ou eget bound, ou eget cale a O (m 1/p 1/ N α/+(p 1)/p T ( ) ). p 1 2/q (23) 13

14 2αp α(p 1)+1 Optimizing ove the exponent of N eult in the choice = which i between 1 and 2 only when α 1. Howeve, multitak poblem ae inteeting when the numbe of tak i vey lage, potentially much lage than the numbe of expet themelve, o that pefoming the tak independently i eally bad. Conide the cae α > 2. If we et = 2 and 1 p 1 = log m, ou eget cale a O (m 1/2 N (α/2+ 1 (1+log m)) ) T (1 + log m) which i vey cloe to the optimal bound. 4.2 Compaion with exiting eult The peviou bet eult on thi poblem ae in the pape of Cavallanti et al[5]. Thi pape decibe a multi-tak p-nom pecepton algoithm, fo which a mitake bound i hown. To compae ou eget bound with thei mitake bound, we fit need to meaue the two algoithm unde the ame lo function. Fo thi, we fit tate an eay eduction to a hinge lo eget bound fo any algoithm that give bounded eget in the expet etup. Lemma 6. Conide any online algoithm that take a equence of loe x t on expet and output a ditibution ove them with the cumulative eget bounded a R(T ). Then thee i an algoithm that ha it eget unde hinge lo bounded by R(T ) in any claification poblem when compaed to all poible weight vecto in the pobability implex. Poof. The poof i a imple eduction that ue the expet algoithm a a black box. Suppoe at time t, ou algoithm ha a ditibution w t ove the expet. We eceive a quey point x t+1 and make a pediction ign(w t x t+1 ). Then we eceive y t+1. If ou pediction i coect, then we pa a lo vecto of all zeo to ou algoithm, othewie we pa the quey point y t+1 x t+1 to it. The eget bound of ou algoithm implie that y t+1 wt x t+1 I(y t+1 wt x t+1 < 1) y t+1 u x t I(y t+1 wt x t+1 < 1) + R(T ) fo any ditibution u ove the expet. Adding T I(y t+1w t x t+1 < 1) to both ide give (1 y t+1 wt x t+1 )I(y t+1 wt x t+1 < 1) (1 y t+1 u x t )I(y t+1 wt x t+1 < 1) + R(T ) The left hand ide i the hinge lo of ou algoithm, while the ight hand ide i an uppe bound on the hinge lo of the compaato, which complete the poof. The eaon why thi lemma i ueful i that we can diectly tanlate ou eget bound to a eget bound unde hinge lo in a claification etup, thu allowing diect compaion to the eult of [5]. The eget bound in that pape cale a O((K m) T log max{k, N}). Putting K = N α, we ee that thi eget cale a O(N α ) which i much woe than a nea optimal eget of oughly O(N α/2 ) achieved by ou algoithm. 5 Concluion In thi pape we have examined a multitak online leaning poblem. The key challenge in thi poblem i that the leane need to infe tak elatedne along the way. We cat thi a a matix egulaization poblem, which lead to two poible appoache baed on tuctual and pectal egulaization. We wok with a tuctual matix nom, that lead to computationally efficient low eget algoithm. The eget bound we obtained ae not optimal, but, a we demontated, get quite cloe to the optimal eget fo ome etting of poblem paamete. The bound eem to be the bet known bound fo any deteminitic and computationally feaible algoithm fo thi poblem. 14

15 Refeence [1] Jacob Abenethy, Pete L. Batlett, and Alexande Rakhlin. Multitak leaning with expet advice. In Poceeding of the Confeence on Leaning Theoy, page , [2] Yonatan Amit, Michael Fink, Nathan Sebo, and Shimon Ullman. Uncoveing haed tuctue in multicla claification. In ICML 07: Poceeding of the 24th intenational confeence on Machine leaning, page 17 24, New Yok, NY, USA, ACM. [3] Andea Agyiou, Theodoo Evgeniou, and Maimiliano Pontil. Convex multi-tak featue leaning. Machine Leaning, [4] Andea Agyiou, Chale A. Micchelli, Maimiliano Pontil, and Yiming Ying. A pectal egulaization famewok fo multi-tak tuctue leaning. In J.C. Platt, D. Kolle, Y. Singe, and S. Rowei, edito, Advance in Neual Infomation Poceing Sytem 20, page MIT Pe, Cambidge, MA, [5] Giovanni Cavallanti, Nicoló Cea-Bianchi, and Claudio Gentile. Linea algoithm fo online multitak claification. In Poceeding of Confeence on Leaning Theoy, [6] Nicolo Cea-Bianchi and Gabo Lugoi. Pediction, Leaning, and Game. Cambidge Univeity Pe, New Yok, NY, USA, [7] Ofe Dekel, Philip M. Long, and Yoam Singe. Online leaning of multiple tak with a haed lo. J. Mach. Lean. Re., 8: , [8] Bian Kuli, Mátyá Sutik, and Indejit Dhillon. Leaning low-ank kenel matice. In ICML 06: Poceeding of the 23d intenational confeence on Machine leaning, page , New Yok, NY, USA, ACM. [9] Alexande Rakhlin. Lectue note on online leaning. Unpublihed lectue note,