Collective Support Recovery for Multi-Design Multi-Response Linear Regression

Transcription

1 IEEE TRANACTION ON INFORMATION THEORY, VOL XX, NO XX, XX 4 Collective upport Recovery for Multi-Deig Multi-Repoe Liear Regreio eiguag ag, Yigbi Liag, Eric P Xig Abtract The multi-deig multi-repoe MDMR liear regreio problem i ivetigated, i which deig matrice are Gauia with covariace matrice Σ : Σ,, Σ for liear regreio tak Deig matrice acro tak are aumed to be idepedet The upport uio of p- dimeioal regreio vector collected a colum of matrix B i recovered uig l /l -regularized Lao ufficiet ad eceary coditio o ample complexity are characterized a a harp threhold to guaratee ucceful recovery of the upport uio Thi model ha bee previouly tudied via l /l -regularized Lao i [] ad via l /l l /l -regularized Lao i [], i which harp threhold o ample complexity i characterized oly for ad uder pecial coditio I thi work, uig l /l -regularized Lao, harp threhold o ample complexity i characterized uder tadard regularizatio coditio Namely, if > c pψb, Σ : logp where c p i a cotat, ad i the ize of the upport et, the l /l -regularized Lao correctly recover the upport uio; ad if < c pψb, Σ : logp where c p i a cotat, the l /l -regularized Lao fail to recover the upport uio I particular, the fuctio ψb, Σ : capture the impact of the parity of regreio vector ad the tatitical propertie of the deig matrice o the threhold o ample complexity Therefore, uch threhold fuctio alo demotrate the advatage of oit upport uio recovery uig multi-tak Lao over idividual upport recovery uig igle-tak Lao I INTRODUCTION Liear regreio i a imple but practically very ueful tatitical model, i which a -ample repoe vector Y ca be modeled a Y X β where X R p i the deig matrix cotaiig ample of feature vector, β β,, β p R p cotai regreio coefficiet, ad R i the oie vector The goal i to fid the regreio coefficiet β uch that the liear relatiohip i a accurate a poible with regard to a certai performace criterio The problem i more iteretig i high dimeioal regime with a pare regreio vector, i which The material i thi paper wa preeted i part at the Iteratioal Coferece o Artificial Itelligece ad tatitic AITAT, cottdale, AZ, UA, April 3 The work of ag ad Y Liag wa upported by NF CAREER Award CCF ad NF CCF The work of Eric P Xig wa upported by NIHRGM87694, FA95547, ad NIHRGM9356 eiguag ag ad Yigbi Liag are with the Departmet of Electrical Egieerig ad Computer ciece, yracue Uiverity, yracue, NY 344 UA {wwag3,yliag6}@yredu Eric P Xig i with the Departmet of Machie Learig, Caregie Mello Uiverity, Pittburgh, PA 53 UA epxig@ccmuedu the ample ize ca be much maller tha the dimeio p of the regreio vector I order to etimate the pare regreio vector, it i atural to cotruct a optimizatio problem with a l -cotrait o β, ie, the umber of ozero compoet of β However, uch a optimizatio problem i ocovex ad i geeral very difficult to olve i a efficiet maer a commeted i [3] More recetly, the covex relaxatio referred to a Lao ha bee tudied with a l -cotrait o β baed o the idea i ome emial work [4] [6] More pecifically, the regreio problem ca be formulated a: mi β R p Y X β l λ β l The l -regularized etimator ha bee proved i [7] to have imilar behavior to Datzig elector, which wa propoed i [8] Variou efficiet algorithm have bee developed to olve the above covex problem efficietly ee a review moograph [9], although the obective fuctio i ot differetiable everywhere due to l -regularizatio Moreover, the l -regularizatio i critical to force the miimizer to have pare compoet a how i [4] [6] A vat amout of recet work ha tudied the high dimeioal liear regreio problem via l -regularized Lao uder variou aumptio For example, the tudie [5], [] [4] ivetigated the oiele ceario ad howed that recovery of true coefficiet could be guarateed with certai coditio o deig matrice ad parity A umber of tudie focued o uig l -regularizatio to achieve parity recovery for oiy ceario ome work eg, [5] [7] focued o the problem with determiitic deig matrice, wherea other work eg, [8], [9] tudied the problem with radom deig matrice The work [] ivetigated liear regreio model via trace orm [] ad [] tudied liear regreio model uig a fuio pealty kow a the total variatioal pealty Geeralized from the l -regularized liear regreio problem which aim at electig variable idividually, group Lao i applied to regreio vector β i the liear regreio model to elect grouped variable eg, [3], [4] The work [5] ad [6] applied group Lao for tudyig empirical rik miimizatio problem The work [7] tudied the leat quare optimizatio problem with group Lao Thi lie of reearch i further geeralized to blockregularizatio for high-dimeioal multi-repoe ie, multitak liear regreio problem, ee, eg, [8], [9] ad referece therei For a multi-tak regreio problem, we

2 IEEE TRANACTION ON INFORMATION THEORY, VOL XX, NO XX, XX 4 have the followig model: Y XB where Y R of which each colum correpod to the output of oe tak, X R p i the deig matrix, the regreio matrix B R p ha each colum correpodig to the regreio vector for oe tak, ad R ha each colum correpodig to the oie vector of oe tak For each colum Y k of the matrix Y, it i clear that Y k X β k k, where β k ad k are the correpodig colum i B ad The each colum i a igle-tak liear regreio problem ad ca be olved idividually However, the idividual problem ie, tak ca alo be coupled together via a block regularized Lao ad olved oitly i oe problem Variou type of block regularizatio have bee propoed ad tudied I the work [9], the l /l -regularizatio wa adopted to recover the upport uio of the regreio vector More pecifically, the followig problem wa tudied mi B R p Y XB F λ B l/l, where la/l b i defied i 7 i ectio II-A ufficiet ad eceary coditio for correct recovery of the upport uio ie, the uio of the upport of all colum of B were characterized Block regularized Lao a well a group Lao ha alo bee applied to tudy variou other model For example, the l p /l q -regularizatio wa tudied for a determiitic ad oiele model i [3] The l /l q -regularized Lao wa adopted for learig tructured liear regreio model i [3] The l /l -regularized Lao wa ued to ivetigate a multi-repoe regreio model i [3], [33] The l /l - regularizatio wa ued for tudyig empirical rik miimizatio problem i [34], multi-tak feature problem i [35], ad multichael pare recovery i [36] The l /l q -regularized Lao wa adopted to aalyze ormal mea model i [37] Blockwie pare regreio wa ued to tudy a geeral lo fuctio i [38] I additio to regularized optimizatio method, greedy algorithm uch a p-threholdig ad p- imultaeou matchig puruit [39] have alo bee tudied I the multi-repoe liear regreio problem give i, the deig matrix i idetical for all tak, ie, X i the ame for all colum vector of Y ad B However, i may applicatio, it i ofte the cae that differet output variable may deped o deig variable that are differet or ditributed differetly Thu, the reultig model iclude liear regreio model with differet deig matrice ad i give by: Y k X k β k k 3 for k,,, where Y k R, X k R p, β k R p, ad k R e refer to the above problem a the multi-deig multi-repoe MDMR liear regreio model, ad the goal i to recover β k for k,, oitly For fixed matrice X,, X, the problem ha bee tudied i [4], [4] via the l /l -regularized Lao ad via a variat of orthogoal matchig puruit i [4] For radom deig matrice, thi model ha bee tudied via l /l -regularized Lao i [] ad via l /l l /l -regularized Lao i [] for icorporatig both row parity ad idividual parity I thi paper, we tudy the MDMR problem for radom deig matrice via l /l -regularized Lao Although thi may eem to oly likely offer expected reult imilar to thoe i [9], [], ad [], our exploratio tur out to provide more iight which were ot captured i previou tudie e dicu thee i depth i ectio I-B I our model, it i aumed that the deig matrice are Gauia ditributed ad are idepedet acro tak Furthermore, the ditributio of deig matrice are alo differet acro tak For each tak k, the row vector of X k i Gauia with mea zero ad the covariace matrix Σ k for k,, The oie vector ad hece the output vector are alo Gauia ditributed ad idepedet acro tak e are itereted i oit recovery of the uio of the upport et ie, the upport uio of regreio vector β, [, β e collect thee vector β together a a matrix B,, ] β e adopt the l /l -regularized Lao problem for recovery of the upport uio via the followig optimizatio problem: mi Y k X k β k λ B B R p l/l 4 k [ β where B,, ] β I thi way, the liear regreio problem are coupled together via the regularizatio cotrait e how that thi approach i advatageou a oppoed to idividual recovery of the upport et for each liear regreio problem Thi i becaue the regreio model may hare their ample i oit upport recovery o that the total umber of ample eeded ca be igificatly reduced compared to performig each tak idividually A Mai Cotributio I the followig, we ummarize the mai cotributio of thi work Our reult cotai two part: the achievability ad the covere, correpodig repectively to ufficiet ad eceary coditio uder which the l /l -regularized Lao problem recover the upport uio for the MDMR liear regreio problem Our proof adapt the techique developed i [8] ad i [9], but ivolve otrivial developmet to deal with the differetly ditributed deig matrice acro tak Thi alo lead to iteretig geeralizatio of the reult i [9] a we articulate i ectio I-B More pecifically, we how that uder certai coditio that the ditributio of the deig matrice atify, if > c p ψb, Σ : logp, where ψ i defied i 8 i ectio II-A ad c p i a cotat, the the l /l -regularized Lao recover the upport uio for the MDMR liear regreio problem; ad if < c p ψb, Σ : logp, where c p i a cotat, the the l /l -regularized Lao fail to recover the upport uio Thu, ψb, Σ : logp erve a a harp threhold o the ample ize I particular, ψb, Σ : capture the parity of B ad the tatitical propertie of the deig matrice, which are importat i determiig the ufficiet ad eceary coditio for ucceful recovery of the upport uio The property of

3 IEEE TRANACTION ON INFORMATION THEORY, VOL XX, NO XX, XX 4 3 ψb, Σ : alo capture the advatage of the multi-tak Lao over olvig each problem idividually via the igletak Lao e how that whe the tak hare the ame upport et although the deig matrice ca be differetly ditributed, ψb, Σ : k ψ β k, Σk Thi mea that the umber of ample eeded per tak for multi-tak Lao to oitly recover the upport uio i reduced by compared to that of igle-tak Lao to recover each upport et idividually O the other had, if the tak have dioit upport et, the ψb, Σ : k ψ β k, Σ k Thi implie that the multi-tak Lao doe ot provide gai i the ample ize eeded per tak for upport recovery compared to igle-tak Lao Betwee thee two extreme cae, tak ca have overlapped upport et with differet overlappig level, ad the impact of thee propertie o the ample ize for recovery of the upport uio i quatitatively captured by ψb, Σ : B Compario to Previou Reult A we metioed before, the MDMR model ha alo bee tudied i [] ad [], i which l /l ad l /l l /l - regularizatio were adopted for upport uio recovery, repectively I thee tudie, harp threhold o ample complexity i characterized oly for ad uder pecial coditio o XkT k X k k I our work, uig l /l -regularized Lao, we are able to characterize the harp threhold uder tadard regularizatio coditio The MDMR model with differetly ditributed deig matrice acro tak ca be viewed a geeralizatio of the multi-repoe model with a idetical deig matrix acro tak tudied i [9] It i thu iteretig to compare our reult to the reult i [9] For the ceario whe the tak hare the ame regreio vector, it i how i [9] that the maor advatage of oitly olvig a multi-tak Lao problem over olvig each igle-tak Lao problem idividually i reductio of effective oie variace by the factor But the ample ize eeded per tak for recovery of the upport uio via multi-tak Lao i the ame a that eeded for recovery of each upport et idividually via igle-tak Lao Thi implie that multi-tak Lao doe ot offer beefit i reducig the ample ize i the order ee for thi cae Our reult, o the other had, how that the beefit i ample complexity by uig multi-tak Lao for recovery of upport uio arie whe the deig matrice are idepedetly ditributed acro tak For uch a cae, the ample ize eeded per tak i reduced by via multi-tak Lao compared to recovery of each upport et idividually via igle-tak Lao Coequetly, our reult i a otrivial geeralizatio of the reult i [9] For the ceario whe the tak have dioit upport et, our reult i coitet with the reult i [9], which ugget that there i o advatage of performig multi-tak Lao a oppoed to performig igletak Lao for each tak C Relatiohip to Joitly Learig Multiple Markov Network Oe applicatio of the MDMR liear regreio model i to oitly learig multiple Gauia Markov etwork tructure I thi cotext, it olve a multi-tak eighbor electio problem Thi i alo a atural ceario, i which feature ad their ditributio vary acro tak e coider Gauia Markov etwork, each with p ode repreeted by X k,, Xk p for k,, The ditributio of the Gauia vector for graph k i give by N, Σ k p, where Σ k p Rp p Aume for each graph, there are iid ample geerated baed o the oit ditributio of the ode The obective i to etimate the coectio relatiohip of ode baed o the ample e deote ample of each variable X k by a colum vector k X R for,, p ad k,, For each graph k ad each ode with idex a, the ample vector X k a ca be expreed a: X k a X k a β k k a 5 where X k a i a p matrix that cotai all colum vector k X for a, β k i a p-dimeioal vector coitig of the etimatio parameter of X a k give X k with a, ad k a i the -dimeioal Gauia vector cotaiig iid compoet with zero mea ad variace give by σ k V arxa CovX a, X, a Cov X, a CovX, a, X a It ha bee how that the ozero compoet of the vector β k repreet exitece of the edge betwee the correpodig ode ad ode a i graph k Hece, etimatio of the upport et of β k provide a etimatio of the graph tructure, which i referred to a the eighbor electio problem i [4] Therefore, multi-tak Lao for the MDMR liear regreio problem provide a ueful approach for oit eighbor electio over graph It i clear that i thi cae, the deig matrice X k a i geeral have differet ditributio acro k, ad hece the MDMR model i well utified e ote that oitly learig multiple graph ha alo bee tudied i [43] ad [44], which adopted a differet obective fuctio of the preciio matrix Σ Via the MDMR liear regreio model, we characterize the threhold-baed ufficiet ad eceary coditio for oit recovery of the graph D Applicatio of the Model I thi ubectio, we dicu practical applicatio of the MDMR model I geeral, uch a model i advatageou if it i applied to the ceario, i which repoe i differet tak deped o their correpodig feature i a imilar way, ad thu the upport overlap acro tak It i typical that feature are ot correlated i the ame way acro tak, which i captured by differet ditributio of the deig matrice acro tak i the model Furthermore, ample of feature are collected from differet idividual acro tak, which i

4 IEEE TRANACTION ON INFORMATION THEORY, VOL XX, NO XX, XX 4 4 captured by idepedetly ditributed deig matrice i the model Oe example of the above ceario i i the cae of aalyzig how gee-expreio deped o geotype The problem iclude multi-tak of liear regreio if we are itereted i multiple group of people with each group havig oe type of dieae Here, it i typical that the gee expreio deped o may geotype i a imilar way acro differet group of people, ad hece the regreio have overlappig upport It i alo typical that geotype are correlated i differet way acro differet group of people due to differet type of dieae Furthermore, data are collected from differet idividual i differet group, ad hece deig matrice have idepedet ditributio acro tak II PROBLEM FORMULATION AND NOTATION I thi paper, we tudy the MDMR liear regreio problem give by 3, which cotai liear regreio Here, the deig matrice X,, X ad oie vector,, are Gauia ditributed, ad are idepedet but ot idetical acro k For each tak k, X k ha idepedet ad idetically ditributed iid row vector with each beig Gauia with mea zero ad covariace matrix Σ k, ad the oie vector k ha iid compoet with each beig Gauia with mea zero ad variace σ k e let σ k σ k I 3, β k deote the true regreio vector for each tak k e defie the upport et for each β k a k : { {,, p} β k } The upport uio over tak i defied to be : k k I thi paper, we are itereted i etimatig the upport uio oitly for tak e adopt the l /l -regularized Lao to recover the upport uio for the MDMR liear regreio model More pecifically, we olve the multi-tak Lao give i 4 ad rewritte below: mi Y k X k β k λ B B R p l/l 6 k [ β where B,, ] β I thi way, the liear regreio problem are coupled together via the regularizatio cotrait I thi paper, we characterize coditio uder which the olutio to the above multi-tak Lao problem correctly recover the upport uio of the true regreio vector for tak A Notatio e itroduce ome otatio that we ue i thi paper For a matrix A R p, we defie the l a /l b block orm a a/b /a p A la/l b : A i b 7 i e alo defie the operator orm for a matrix a A a,b : up Ax a x b I particular, we defie the pectral orm a A A, ad the l -operator orm a A A,,,p k A k, which are pecial cae of the operator orm [ β For matrix B,, ] β that appear i 6, β k deote it kth colum for k,, e further [ let B i to be the ith row of B imilarly, for B β,, ] β that cotai true regreio vector, it kth colum i deoted by β k ad the ith row i deoted by Bi e ext defie the ormalized row vector of B a B i if B Zi B i i l otherwie, ad defie the matrix Z to cotai Zi a it ith row for i,, p To avoid cofuio, we ue B to deote the olutio to the multi-tak Lao problem 6 The upport uio B for a matrix B R p i deoted a B {i {,, p} B i }, which iclude idice of the ozero row of the matrix B e ue to repreet B ie, the true upport uio for coveiece ad ue c to deote the complemet of the et e let deote the ize of the et For ay matrix X k R p, the matrix X k cotai the colum of matrix Xk with colum idice i the et, ad X k cotai the colum of matrix c X k with colum idice i the et c imilarly, B ad Z repectively cotai row of B ad Z with idice i A each row of matrix X k i Gauia ditributed a N, Σ k, we ue Σ k to deote the covariace matrix for each row of X k, ad ue Σk c to deote the cro covariace betwee row of X k ad c Xk For coveiece, we ue Σ : to deote a et of matrice Σ,, Σ e alo defie the followig fuctio of matrice Q : to implify our otatio: ρ u Q : : ρ l Q : : mi c k Qk, mi i, c, i k [ Q k Qk ii I particular, our reult cotai the fuctio ρ u ad ρ l Σ : c c, where Σ k c c each row of X k with c Xk give For matrix B, we defie b mi ] Q k i Σ : c c i the covariace matrix of B l e mi defie the followig fuctio that capture the parity of B ad the tatitical propertie of the deig matrice X : : ψb, Σ : : Z T Z k Σ k k, 8 k where Z k i the kth colum of Z e ote that thi defiitio of ψ fuctio i differet from the previou work [9] with the ame deig matrix for all tak Here, due to differet deig matrice acro the tak, ψ deped o quatitie Z Z T k Σ k k with each depedig o a colum vector Z k

5 IEEE TRANACTION ON INFORMATION THEORY, VOL XX, NO XX, XX 4 5 g e deote g o f if lim f, ad g g O f if lim f c o, where the cotat < c o < III MAIN REULT I thi ectio, we provide our mai reult o uig l /l - regularized Lao to recover the upport uio for the MDMR liear regreio model Our reult cotai two part: oe i the achievability, ie, ufficiet coditio for the l /l - regularized Lao to recover the upport uio; ad the other i the covere, ie, coditio uder which the l /l -regularized Lao fail to recover the upport uio e the dicu implicatio of our reult by coiderig a few repreetative ceario, ad compare our reult with thoe for the multivariate liear regreio with a idetical deig matrix acro tak A Achievability ad Covere e firt itroduce a umber of coditio o covariace matrice Σ k for k,,, which are ueful for the tatemet of our reult C There exit a real umber γ, ] uch that A γ, where A k Σ k c Σ k for c ad C There exit cotat < C mi C < uch that all eigevalue of the matrix Σ k are cotaied i the iterval [C mi, C ] for k,, C3 There exit a cotat D < uch that k Σ k D e ote that the above aumptio C-C3 are atural geeralizatio of a umber of claical aumptio appearig Σ k i the codi- i previou work The matrix Σ k c tio C capture correlatio betwee X k ad X k c k,,, Hece, a upper boud o the orm A guaratee ufficiet ditictio betwee X k ad X k, c which i eceary for the coitecy of upport recovery imilar coditio were propoed to guaratee igal recovery for the liear regreio problem, which are referred to a bouded mutual coherece i [5], [8], [45], a correlatio coditio i [46], ad a irrepreetable coditio i [6], [9] The coditio C i imilar to the retricted eigevalue coditio i [7], the retricted iometry property RIP i [47] ad oe coditio i [6] The coditio C3 require certai coherece level withi the feature colum of X uch that the upport feature are highly correlated i order for correct upport recovery a commeted i [9] e refer to [48] for a ummary of the coditio ad their relatiohip for performace guaratee i liear regreio problem I thi paper, we coider the aymptotic regime, i which p,, ad log p I uch a regime, we itroduce the coditio o the regularizatio parameter ad the ample ize a follow: for fp log p P Regularizatio parameter λ, where the fuctio fp i choe uch that fp a p, fp log p ad a, ie, λ a P Defie ρ,, λ a 8σ ρ,, λ : log λ D C mi C mi ad require ρ,,λ b o mi The followig theorem characterize ufficiet coditio for recovery of the upport uio via l /l -regularized Lao Theorem Coider the MDMR problem i the aymptotic regime, i which p, ad logp e aume that the parameter, p,, B, Σ : atify the coditio C-C3, ad P-P If for ome mall cotat v >, > vψ B, Σ : ρ u Σ : logp c c γ, 9 the the multi-tak Lao problem 6 ha a uique olutio B, the upport uio B i the ame a the true upport uio B, ad B B l /l ob mi with the probability greater tha exp c log exp c log p where c ad c are cotat Theorem provide ufficiet coditio o the ample ize uch that the olutio to the l /l -regularized Lao problem correctly recover the upport uio of the MDMR liear regreio model e ext provide a theorem about the coditio o the ample ize uder which the olutio to the l /l -regularized Lao problem fail to recover the upport uio Theorem Coider the MDMR problem i the aymptotic regime, i which p, ad logp e aume that the parameter, p,, B, Σ : atify the coditio C-C ad the coditio: / o ad λ If for ome mall cotat v >, ρ l Σ : < vψb, Σ : log p c c the with the probability greater tha exp c c 3 exp c 4 γ, for ome poitive cotat c, c 3 ad c 4, o olutio B to the multi-tak Lao problem 6 recover the true upport uio ad achieve B B l /l ob mi The proof of Theorem ad are provided i ectio V ad V I, repectively e ote that baed o the defiitio, ρ u Σ : c c i 9 ad ρ l Σ : c c Σ : c c i deped o the etry value of, which do ot chage with the ytem parameter, p, The oly poibility that the two quatitie ca cale

6 IEEE TRANACTION ON INFORMATION THEORY, VOL XX, NO XX, XX 4 6 with i due to the imum ad miimum over the etrie of the matrix However, thi typically doe ot caue order level differece i the two quatitie I term of the phyical meaig, ρ u Σ : c c capture the larget expected value of feature vector i c, ad ρ l Σ : c c capture the mallet expected value of the differece betwee ay two feature vector I mot practical ceario, it i reaoable to aume that thee two quatitie do ot cale a ytem dimeio icreae, ad their value hould be bouded Therefore, combiig Theorem ad, it i clear that the quatity ψb, Σ : logp erve a a threhold o the ample ize, which i tight i the order ee A the ample ize i above the threhold, the multi-tak Lao recover the true upport uio, ad a the ample ize i below the threhold, the multi-tak Lao fail to recover the true upport uio The followig propoitio provide boud o the calig behavior of the fuctio ψb, Σ : i the aymptotic regime Propoitio Coider the MDMR liear regreio model with the regreio matrix B ad the covariace matrice Σ : atifyig the coditio C, the fuctio ψb, Σ : i bouded a ψb, Σ : C C mi The proof of the Propoitio i provided i Appedix A I the ext ubectio, we explore the propertie of the quatity ψb, Σ : i order to udertad the impact of parity of B ad covariace matrice Σ : o ample complexity for recoverig the upport uio B Implicatio The quatity ψb, Σ : capture parity of B ad tatitical propertie of deig matrice Σ :, ad hece play a importat role i determiig the coditio o the ample ize for recovery of the upport uio a how i Theorem ad I thi ectio, we aalyze ψb, Σ : for a umber of repreetative cae i order to udertad advatage of multi-tak Lao which olve multiple liear regreio problem oitly over igle-tak Lao which olve each liear regreio problem idividually e deote ψ β k, Σ k a the fuctio correpodig to a igle liear regreio problem, where β k repreet the kth colum of B It i clear that ψ β k, Σ : capture the threhold o the ample ize for the igle-tak Lao problem Compario of ψb, Σ : ad ψ β k, Σ k provide compario betwee multi-tak Lao ad igle-tak Lao i term of the umber of ample eeded for recovery of the upport uio/et e explicitly expre ψb, Σ : ad ψ β k, Σ k a follow: ψb, Σ : Bik B k k B i i B l l Σ k i 3 ψ β k, Σ k k k β i β i β k β k i Σ k i 4 where Bik deote the i, kth etry of the matrix B ad k β i deote the ith etry of the vector β k e firt tudy the ceario, i which all tak have the ame regreio vector, ad hece have the ame upport et Propoitio Idetical Regreio Vector If B ha idetical colum vector, ie, β k β for k,,, the ψb, Σ : ψ β, Σ k 5 k Proof Uder the aumptio of the propoitio, B β T, where β R p Hece, Z k ig β, where the vector β cotai compoet i the upport ψb, Σ : Z T Z k Σ k k k ig β T k Σ k ig β k ψ β, Σ k 6 Remark Propoitio implie that the umber of ample per tak eeded to correctly recover the upport uio via multi-tak Lao i reduced by a factor of compared to igle-tak Lao that recover each upport et idividually It ca be ee that although the tak ivolve deig matrice that have differet covariace, a log a depedece of the output variable o the feature variable i the ame for all tak, the tak hare ample i multi-tak Lao to recover the upport uio o that the ample ize eeded per tak i reduced by a factor of Hece, there i a igificat advatage of groupig tak with imilar regreio vector together for multi-tak learig Propoitio ca be viewed a a geeralizatio of the reult i [9], i which the deig matrice for the tak are the ame The reult i [9] ugget that if the tak hare the ame regreio vector, there i o beefit i term of the umber of ample eeded for upport recovery uig multitak Lao compared to igle-tak Lao Our reult ugget that the beefit of multi-tak Lao i fact arie whe the deig matrice are differetly ditributed For uch a cae, we how that the ample ize eeded per deig matrix ie, per tak i reduced by the factor Moreover, compared to recovery of each upport et idividually via igle-tak Lao, multi-tak Lao alo reduce ample ize per tak by the factor However, uch a advatage doe ot appear if the tak have the ame deig matrix ad regreio vector a i [9] e ext tudy a more geeral cae whe regreio vector are alo differet acro tak but the upport et of tak are the ame i additio to varyig deig matrice acro tak

7 IEEE TRANACTION ON INFORMATION THEORY, VOL XX, NO XX, XX 4 7 Propoitio 3 Varyig Regreio Vector with ame upport uppoe Bk for all ad k,,, ad all regreio coefficiet are bouded, ie, Bk k Bk B k k, where B k B k ad k > i a mall perturbatio cotat with B k > k If Bik B k Σ k for all i, ad k,,, the i ψb, Σ : k ψ β k, Σ k k Bk k Bk k Proof Baed o the aumptio for B, we obtai the followig upper boud o ψb, Σ : ad lower boud o ψ β k, Σ k : ψb, Σ : k Bk k ψ β k, Σ k Bk k i i B ikb k Combiig the above boud, we obtai ψb, Σ : k ψ β k, Σ k B ikb k Σ k ; i 7 Σ k 8 i k Bk k Bk k Propoitio 3 i a tregtheed verio of Propoitio i that Propoitio 3 allow both the regreio vector ad deig matrice to be differet acro tak ad till how that the umber of ample eeded i reduced by a factor of compared to igle-tak Lao, a log a the upport et acro tak are the ame Propoitio 4 Dioit upport et uppoe that the upport et k of all tak are dioit Let k k, ad hece k k The, ψb, Σ : ψ β k, Σ k k Proof By the aumptio of the propoitio, we obtai: ψb, Σ : β ig k k T Σ k β k ig k ψ β k, Σ k 9 Propoitio 4 ugget that if the tak have dioit upport et for regreio vector, the advatage of the multi-tak Lao vaihe Thi i reaoable becaue the tak do ot beefit from harig the ample for recoverig the upport if their upport et are dioit The eetial meage of Propoitio 4 hould ot chage if the tak have differet deig matrice ad/or differet regreio vector The critical aumptio i Propoitio 4 i the dioit upport et uch a behavior i alo demotrated i the umerical imulatio a i Fig 7, i which the ample ize eeded per tak for upport recovery i multi-tak Lao almot equal the ample ize eeded i igle-tak Lao Corollarie ad 4 provide two extreme cae whe the tak hare the ame upport et ad have dioit upport et, repectively The umber of ample eeded per tak for recovery of the upport uio goe from / of to the ame a the ample ize eeded for igle-tak Lao Betwee thee two extreme cae, tak may have overlapped upport et with variou overlappig level Correpodigly, the umber of ample eeded for recoverig the upport uio hould deped o the overlappig level of the upport et ad i captured preciely by the quatity ψb, Σ : e demotrate uch behavior via our umerically reult i the ext ectio IV NUMERICAL REULT I thi ectio, we provide umerical imulatio to demotrate our theoretical reult o uig block-regularized multitak Lao for recovery of the upport uio for the MDMR liear regreio model e tudy how the ample ize eeded for correct recovery of the upport uio deped o parity of the regreio vector, o the ditributio of the deig matrice, ad o the umber of tak Experimet e firt tudy the ceario coidered i Propoitio whe the tak have the ame regreio vector, ie, B β T e et β, where i the commo upport et acro tak e et the covariace matrix Σ k to be tridiagoal ad differet acro tak a follow For Σ k, a how below, all etrie i the mai diagoal have the ame value C a, etrie i the immediate upper ad lower diagoal take the ame value C k b, which varie with the tak idex k C a C k C k b b C a C k b C k b C a p p I our experimet, we chooe C a ad C k b /k for k,,, It i eay to check that thi matrix i poitive emi-defiite The parity of liear regreio vector i liearly proportioal to the dimeio p, ie, αp, with the parameter α cotrollig the parity of the model e et α /8 e chooe the dimeio p 8, 56, 5 e et the regularizatio parameter λ 4 log p log / e olve the l /l -regularized multi-tak Lao problem 6 for recovery of the upport uio for,, 4, 8 Fig plot the empirical probability ie, empirical frequecy of correct recovery of the upport uio a a fuctio A tridiagoal matrix i a matrix that ha ozero elemet oly o the mai diagoal, ad the firt diagoal below ad above the mai diagoal

8 IEEE TRANACTION ON INFORMATION THEORY, VOL XX, NO XX, XX 4 8 p 8 p /[ logp ] p /[ logp ] p /[ logp ] p /[ logp ] p /[ logp ] /[ logp ] Fig Impact of umber of tak o the ample ize for ceario with idetical regreio vector ad varyig ditributio for deig matrice acro tak Fig Impact of umber of tak o the ample ize for ceario with o-equal regreio value ad idetical deig matrix ditributio acro tak of the caled ample ize It ca be ee that the ample ize for guarateeig correct recovery cale i the order of logp for all plot Moreover, a the umber of tak icreae, the ample ize per tak eeded for correct recovery decreae iverely proportioally with, which i coitet with Propoitio Thee reult demotrate that whe the regreio vector are the ame acro tak, multi-tak Lao ha a great advatage compared to igletak Lao ie, i term of reductio i the ample ize eeded per tak Experimet e are alo itereted i the ifluece of uequal regreio value o the ample ize for correct recovery Our ext experimet i take for the ceario i which all tak hare the ame upport et but have uequal regreio value acro tak e cotruct regreio vector with periodic etry value ad with 8p To avoid trivial variatio from Experimet, we ue two regreio value alteratively for each regreio vector Therefore, all regreio vector are ot equal to each other but with all vector havig peroid 6 More pecifically, for k,,, β k 6 k for 6t pe, ad β k 6 k for 6tpe 8, where t pe i ay oegative iteger uch that p The covariace matrice Σ k are et to be idetical acro all tak All covariace matrice take the form of but with C a ad C k b / Other parameter are choe to be the ame a the experimet i Fig Fig plot how the empirical probability of correct recovery chage with the ample ize for p 8, 56, 5 It exhibit the ame behavior a Fig, although ow the regreio vector have uequal value acro tak I particular, it ca be ee that the ample ize eeded for correct recovery decreae a the

9 IEEE TRANACTION ON INFORMATION THEORY, VOL XX, NO XX, XX 4 9 umber of tak icreaed, demotratig the advatage of multi-tak Lao p 8 probability of correct recovery chage with the caled ample ize It ca be ee from the figure that the ample ize eeded for the two-tak Lao i half of the igle-tak Lao Thi demotrate imilar performace to Experimet eve whe the regreio coefficiet are ubtatially differet Orthogoal /[ logp ] p 56 Orthogoal /[ logp ] p 5 Orthogoal /[ logp ] Fig 3 Compario of two-tak Lao with orthogoal regreio vector ad idetical upport ad igle-tak Lao p 8 Idetical upport Overlap upport Dioit upport /[ logp ] p 56 Idetical upport Overlap upport Dioit upport /[ logp ] p 5 Idetical upport Overlap upport Dioit upport /[ logp ] Experimet 3 I Experimet, variatio of differet regreio vector are ot igificat It i iteretig to coider the ceario with the regreio vector ubtatially differet from each other but till with the ame upport I Experimet 3, we tudy uch a cae I particular, we et ie, two tak, β for the firt tak, ad β / [, ] T for the ecod tak It i clear that two regreio vector are orthogoal The covariace matrice for the two tak take the ame form of with C a ad C k b / e alo compare the performace to the igle-tak cae, i which the regreio vector take the form of β, ad the covariace matrix i the ame a the multi-tak cae All other parameter are et to be the ame a Experimet Fig 3 plot how the empirical Fig 4 Impact of overlappig level of upport et o the ample ize with ame regreio value for overlappig etrie ad idetical ditributio for deig matrice acro tak Experimet 4 e ext tudy how the overlappig level of the upport et acro tak affect the ample ize for correct recovery of the upport uio e et, ie, two tak, ad tudy three overlappig model for the two tak: ame upport et { p : 8t pe, where iteger t pe }; dioit upport et φ i which { p : 6t pe, where iteger t pe } ad { p : 6t pe, where iteger t pe }; 3 overlappig upport et i which { p : 4t pe or 4t pe where iteger t pe }, ad { p :

10 IEEE TRANACTION ON INFORMATION THEORY, VOL XX, NO XX, XX 4 4t pe or 4t pe 3 where iteger t pe } e chooe the liear parity model with α /8 e et p 8, 56, 5, ad Σ k I p for k ad e alo et λ 4 log p log / Fig 4 compare the empirical probability of correct recovery a a fuctio of the caled ample ize for the three overlappig model It ca be ee that the model with the ame upport et require the mallet ample ize, ad the model with dioit upport et require the larget ample ize The model with overlappig upport et eed the ample ize betwee the two extreme model Thi i reaoable becaue a the upport et overlap more, tak hare more iformatio i ample for upport recovery ad hece eed le umber of ample for correct recovery The Experimet 4 i take for the cae whe the deig matrice of the two tak have the ame covariace matrix ad the regreio vector are idetical o overlappig etrie It i iteretig to ivetigate how o-equal value i regreio vector ad differet covariace matrice acro the two tak affect the ample complexity e tudy thee ceario i Experimet 5 ad 6 Experimet 5 e tudy the cae whe the regreio vector of the two tak do ot have the ame value o the overlappig etrie For the cae whe the two tak have the ame upport et, we let β k k 8 for 6t pe, ad β k k 8 for 6tpe 8, where iteger t pe uch that p for k, For the overlappig model, ad are the ame a the Experimet 4 For k, β k if 4t pe, ad k β 8 if 4tpe, where iteger t pe uch that p For k, β k 8 if 4t pe, ad β k if 4t pe 3, where iteger t pe uch that p For the dioit cae, the regreio vector are the ame a the Experimet 4 ice o overlappig exit i the dioit model Other parameter Σ :,, p,, λ are kept the ame a the Experimet 4 Fig 5 plot the empirical probability of correct recovery of the upport uio veru the caled ample ize for thi experimet It ca be oberved that Fig 5 exhibit ame behavior a Fig 4 ad demotrate that higher overlappig level acro two tak lead to maller ample ize eeded for recovery, although the regreio vector do ot match value for the overlappig etrie e alo ote that more careful compario of Fig 5 ad Fig 4 ugget that the model with perturbatio o overlappig etrie i regreio vector require a lightly larger ample ize tha the model without perturbatio Experimet 6 e ext tudy how the varyig covariace matrice acro the two tak ifluece the reult e et the covariace matrice Σ k to be differet for k, a follow Other parameter B,, p,, λ are the ame a the experimet i Fig 4 Fig 6 compare the empirical probability of correct recovery veru the caled ample ize for the three overlappig model uder the varyig covariace matrice but the ame value for overlappig regreio etrie acro the two tak The behavior i imilar to that i Fig 4 ad Fig p 8 Idetical upport Overlap upport Dioit upport /[ logp ] p 56 Idetical upport Overlap upport Dioit upport /[ logp ] p 5 Idetical upport Overlap upport Dioit upport /[ logp ] Fig 5 Impact of overlappig level of upport et o the ample ize with o-equal regreio value for overlappig etrie ad idetical covariace matrice acro tak More careful compario of Fig 6 ad Fig 4 ugget that the varyig covariace matrice acro the two tak require larger ample ize tha the cae with idetical covariace matrice Thi i becaue γ i the cae with varyig covariace matrice i maller tha that i the cae with idetical covariace matrice, ad ψ fuctio i larger i the cae with varyig covariace matrice Experimet 7 I thi experimet, we compare the performace of multi-tak Lao with regreio vector havig dioit upport et acro tak with that of igle-tak Lao More pecifically, we tudy a two-tak ceario with dioit upport et { p : 6t pe, where iteger t pe } ad { p : 6t pe, where iteger t pe }, repectively The covariace matrix i the ame a that i Experimet 3 e compare uch a cae with the igle-tak ceario i Experimet 3 Fig 7 plot the empirical probability

11 IEEE TRANACTION ON INFORMATION THEORY, VOL XX, NO XX, XX 4 p 8 p Idetical upport Overlap upport Dioit upport /[ logp ] p Dioit upport /[ logp ] p Idetical upport Overlap upport Dioit upport /[ logp ] p Dioit upport /[ logp ] p Idetical upport Overlap upport Dioit upport /[ logp ] 8 6 Dioit upport /[ logp ] Fig 6 Impact of overlappig level of upport et o the ample ize with ame regreio value for overlappig etrie ad varyig covariace matrice acro tak of correct recovery a a fuctio of caled ample ize It i clear from the figure that the ample ize eeded per tak for multi-tak Lao with dioit upport are almot the ame a that for igle-tak Lao, demotratig that the advatage of multi-tak Lao vaihe if the regreio vector have dioit upport et acro tak V PROOF OF THEOREM Our proof applie the framework developed i [8] ad i [9] baed o the idea of primal-dual wite However, for the MDMR model, we eed to develop ovel adaptatio due to varyig deig matrice acro tak I [9], ice the model ca be expreed by a matrix operatio o regreio matrix, the proof ivolve may operatio for matrice, for which propertie/boud for matrice ca be applied However, the MDMR model i expreed by operatio o idividual Fig 7 Compario of two-tak Lao with dioit upport et ad igletak Lao regreio vector The proof motly ivolve firt maipulatig/boudig idividual regreio vector ad the itegratig thee maipulatio/boud together for coditio acro all tak Our adaptatio eed to make boud i both tep a tight a poible i order to develop harp threhold coditio e ext preet our proof i detail The obective fuctio i the multi-tak Lao problem give i 6 i covex, ad hece the followig aruh- uh-tucker T coditio i ufficiet ad eceary to characterize a optimal olutio: B fb λ Z where fb k Y k X k β k, ad Z B l/l Before itroducig the ufficiet coditio, we firt preet the followig lemma which provide a importat property

12 IEEE TRANACTION ON INFORMATION THEORY, VOL XX, NO XX, XX 4 about the optimal olutio to the above problem Lemma uppoe there exit a optimal olutio B to the multi-tak Lao problem give i 6 uppoe Ẑ i i the ubdifferetial of B l/l at B, ad atifie the T coditio i oitly with B uppoe that Ẑ atifie ẐΩ <, where ẐΩ deote the ubmatrix that cotai l /l row of Ẑ with idice i the et Ω The ay optimal olutio B to 6 mut atify B Ω The proof of Lemma i imilar to that of Lemma i [8] For completee of our paper, we provide the proof of Lemma i Appedix B e ow cotruct a pair B, Ẑ that atify the T coditio i e firt let B be a optimal olutio to the followig optimizatio problem: B argmi B [ fb B c λ B l/l ] ad let Ẑ be the aociated elemet i the ubdifferetial of B l/l uch that B, Ẑ atify the T coditio for the optimizatio problem give i e the let ˆB c, ad let Ẑ c be a elemet i the ubdifferetial of B c l /l that atifie the T coditio oitly with B c for the followig problem argmi B c [fb B B λ B c l/l ] 3 uch Ẑc mut exit if the T coditio for the optimizatio problem 3 implie Ẑ l /l c Now it i eay to ee that B, Ẑ obtaied above atifie the T coditio i ad i hece a optimal olutio tothe problem 6 Furthermore, followig Lemma, if Ẑ l /l c <, the ay optimal olutio B to 6 atifie B c Therefore, coditio Ẑ l /l c < guaratee both that there exit a optimal olutio with the tructure decribed a above ad that all optimal olutio B atifie B c Furthermore, the coditio Ẑ l /l c < guaratee uiquee of the optimal olutio The argumet follow from the proof of Lemma i [8] e ext proceed to characterize the coditio that guaratee Ẑ l /l c < For c ad k,,, we have Ẑ k k T X Π k λ I k k T k Σk X X Z k, 4 where X k deote the th colum of the matrix X k, Σ k T X k X k Σk X k Xk, ad Πk The tep to obtai the above Ẑk i provided i Appedix C for completee Aalyi of V c: e let V Ẑ,, Ẑ e eed to characterize the coditio o that V l < for all c T with high probability e write V ito three term a follow V E V X : }{{} T E V X :, : E V X : }{{} T V E V X :, : }{{} T 3 X : X 5 where,, X ad :,, e ext evaluate T, T, ad T 3 oe by oe Evaluatio of T : By the defiitio of Ẑ, we have the followig coditioal idepedecie: k X k Z k X k Z k X k X : X : X :,,, : 6 Give the above idepedece propertie, we firt derive X : E Ẑk λ E E X k Σ k Σ k X k T : X Π k Σk T : X X k Z X : E k I k E E Z k X : 7 for c, where Σ k repreet the covariace betwee a compoet i X k ad a row i X k e the obtai the followig boud o T l with the proof provided i Appedix D: where A a k Σ k c a e hece obtai c T l T l A a, 8 a Σ k a for c ad A a A γ c a Evaluatio of T : Due to the idepedecy

13 IEEE TRANACTION ON INFORMATION THEORY, VOL XX, NO XX, XX 4 3 Z k X k X :,, : we obtai X : E Ẑk, : λ E E X k X k T : X, : Π k I T X :, : X k Σk k Z E k X :, : Σ k Σ k Z k 9 where the ecod equality follow becaue Z k i a fuctio of X : ad : e the obtai X : E Ẑk, : X : E Ẑk Z Z Σ k Σ k k E k X : 3 Thu, followig from tep imilar to thoe i Appedix D, we obtai T l A a Ẑ EẐ X :, 3 l /l ad hece c T l a A a Ẑ EẐ X : c a l /l A Ẑ EẐ X : l /l γ Ẑ EẐ X : l /l γ Ẑ Z l /l [ γe Ẑ Z X : l /l ] 3 e ext provide the followig lemma give i [9], which i ueful for our proof Lemma [9] Coider the matrix R with row i : B i B i B If i l /l <, the Ẑ Z l /l 4 l /l By applyig the above lemma, give the coditio that we will how later, we obtai l /l < [ c T l 4 γ l /l E l /l X : e will how later i the aalyi of U that l /l i of order o with high probability, ad hece the above iequality hold with high probability ] Evaluatio of T 3 : e itroduce the vector D k uch that Ẑ k k T X Π k λ I k k T k Σk X X Z k X k T D k 33 It i clear that for c, Cov X k X :, : Σ k c c I Uder the coditio that X : ad : are give, we have Ẑk X :, : E [Ẑk X :, :] where N, σ k 34 σ k Σ k c c λ Z T Σ k c c k Σk Z k kt Π k I k 35 Give X :,, : Ẑ k i idepedetly ditributed acro k for k,, Hece, Ẑ k E [Ẑk X :, :] d σ k ξ k give X :, : 36 where ξ k N, i idepedetly ditributed acro k for k,, Thu, [ X : V E V, :] d σkξ k l k give X :, : 37 e hece obtai T 3 d c l c give k σ kξ k c k σ k c k ξ k X :, : 38 e ext provide a ueful boud for χ radom variable, which wa give i [9] Lemma 3 [9] Let Z be a cetral χ ditributed radom variable with the degree d The for all t > d, we have [ ] d P Z t exp t t

14 IEEE TRANACTION ON INFORMATION THEORY, VOL XX, NO XX, XX 4 4 Applyig the above lemma, we obtai for all t >, P > t c k p P ξ k k ξ k > t [ ] p exp t 39 t By applyig the boud o σk together with 39, we further have T 3 c l tρ u Σ : ψb, Σ : c c with the probability larger tha exp 4 exp 4 exp log log [ ] p exp t t [ ] exp 5 5 for t >, where derived i appedix E Γ 6 l /l C mi l /l C mi Γ σk λ 4 For large eough, Γ coverge to zero with a order o e alo ote that ψb, Σ : ha a order O baed o Propoitio I 4, we et t v δ log p where v > ad δ v/3v 4 e ca the how that if > vψ B, Σ : ρ u Σ : logp c c γ, the T 3 l < γ 43 c with the probability larger tha exp 4 exp 4 exp log log exp v log p [ ] exp It follow from 5 that V l T l T l T 3 l Combiig the above equatio with the evaluatio for T, T, T 3, we coclude that V l < with high probability due to 44 ad 57 Aalyi of U : e have obtaied the ufficiet coditio for the exitece ad uiquee of a optimal olutio to the problem give i 6, which guaratee B c It remai to characterize coditio uch that all row of B are ozero ad hece B recover the true upport uio I order to guaratee that every row of B i ozero, it uffice to guaratee that where Each colum U k U k U l /l b mi U B B : k β Σk i give by β k XkT It uffice to guaratee that [ U U k l ] U k λ Z k 45 b mi, for k,, I order to boud U k, we defie l ad hece U k k Σk Σk e the obtai the followig boud U k l X kt k, k λ Σk Z k k l Σk } {{ } T k Σk λ Z k } {{ l } T k 46 e ext evaluate the boud o the two term T k ad T k, repectively

15 IEEE TRANACTION ON INFORMATION THEORY, VOL XX, NO XX, XX 4 5 Evaluatio of T k : e firt derive the followig boud Σk Z k l i a Σk Σk Σ k D i Σk Σk Σ k Σ k b D 47 C mi with probability larger tha exp 3 exp 4 I the above derivatio, tep a follow from the aumptio of the theorem ad A A for A R, ad tep b applie the boud give i 97 i Appedix F Therefore, T k λ D 48 C mi with probability larger tha exp 3 exp 4 Evaluatio of T k : e firt have kt k X : E E σ k Σk XkT k kt X k Σk X : I 49 which implie that give X :, k ha iid compoet with each beig Gauia ditributed a N, σ w k Hece, give X :, we have T k Σk σ C mi with probability larger tha exp k l ξ 5 4, where σ k σ k, ad ξ i the tadard Gauia radom variable The ecod iequality i the precedig derivatio follow becaue A A for A R, ad from the boud 95 provided i Appedix F By applyig Lemma 3 with d, we have P ξ t [ ] exp t 5 t By ettig t log i the above boud, we the obtai t T k σ C mi 8 log σ 5 C mi with the probability larger tha exp 4 exp log log 53 Combiig the boud o T k ad T k, we obtai 8 log σ λ D l C mi U k C mi ρ,, λ 54 with the probability larger tha exp 4 exp 4 exp log log 55 ρ,,λ b mi Thu, the aumptio o guaratee that U k b mi for ufficietly large l Furthermore, we derive the followig boud B B l /l l /l mi B U l /l b mi k U k U k b mi b mi U k k b mi l k ρ,, λ b mi o 56 with the probability larger tha exp 4 exp 4 exp log log 57 ummarizig the aalyi of V c ad U, we coclude that the multi-tak Lao problem give i 6 ha a uique olutio B, whoe upport uio recover the true upport uio B with high probability uder the aumptio of the theorem VI PROOF OF THEOREM Our proof adapt ad further develop the proof techique etablihed i [9] due to varyig deig matrice acro tak Followig from the proof i ectio V, it ca be how that if either Ẑ l /l c > hold or B B ob l /l mi doe ot hold, o olutio B to the multi-tak Lao problem give i 6 recover the correct upport uio ad atifie B B ob l /l mi Hece, if B B l /l ob mi doe ot hold, it i already the cae that the multitak Lao doe ot provide the deired olutio The the

16 IEEE TRANACTION ON INFORMATION THEORY, VOL XX, NO XX, XX 4 6 followig proof i to idetify ufficiet coditio uch that V c l /l > whe B B ob l /l mi hold, where V Ẑ,, Ẑ for c e ue the decompoitio i 5, which i rewritte below: V T T T 3 However, we are ow itereted i lower boudig V c l /l e firt boud thi quatity a follow: V c l /l T c 3 l /l T c l /l T c l /l By the aumptio of the theorem, T c l /l γ e ext coider T c Due to 3, we have T c l /l γ Ẑ Z l /l [ ] γe Ẑ Z X : 58 l /l By the aumptio that B B ob l /l mi hold, followig the proof i ectio V, T c l /l o hold It the uffice to guaratee that T c 3 l /l > γ e recall from 38 that T d 3 c l σ c k ξ k k give X :, :, 59 where ξ k N, are idepedetly ditributed acro k e let V : T c 3 l /l, ad the remaiig part of the proof i to derive a lower boud o V, which take everal tep The firt tep i to how that V i cocetrated aroud it expectatio whe X :, : are give Lemma 4 For ay δ >, [ P V EV δ X : δ, :] 4 exp 6 ρ u Σ : c c k M k Proof e firt cotruct the followig fuctio g : R p R gξ : c k σ k ξ k where ξ k i the etry of the matrix ξ with the idex pair {, k} To explore the cotiuity property of the cotructed fuctio g, we let u u k, c, k,, ad v v k, c, k,, be two matrice e derive the followig boud give X :, : gu gv σ c k u k σ c k v k k k σ c k u k σk v k k k a σ c k u k c v ρ u Σ : c c M k u v F, 6 k where a follow by takig quare o both ide ad comparig variou cro term Therefore, the fuctio g i Lipchitz cotiuou with cotat L ρ u Σ : c c k M k The proof complete by applyig Gauia cocetratio iequality give below for a tadard Gauia vector X ad the Lipchitz fuctio g with the cotat L: P [ gx EgX δ] 4 exp δ /L The ecod tep i to fid a lower boud o E[V ] Lemma 5 For ay fixed δ ad ufficietly large p, the followig iequality hold: [ X : E V, :] Mk δ ρ l Σ : c c log p / k Proof The proof i uder the aumptio that X :, : are give Defie η k Σ k c c ξ k ad therefore, η k N, Σ k c c σk ξ k k k M k ηk k e the have M k Σ k c c ξ k M k η k 6 where k arg k Mk ithout lo of geerality, let k [ X : E V, :] M k E η 63 c The proof complete by applyig the lower boud of E c η It ca be how that E [ η i η ] ρ l Σ : c c E [ ξ i ξ ]