SIGNAL estimation has been a fundamental problem in a

Transcription

1 90 IEEE TRANSACTIONS ON SIGNAL AND INFORMATION PROCESSING OVER NETWORKS, VOL., NO. 3, SEPTEMBER 06 A Olie Parallel Algorihm for Recursive Esimaio of Sparse Sigals Yag Yag, Member, IEEE, Marius Pesaveo, Member, IEEE, Megyi Zhag, Member, IEEE, ad Daiel P. Palomar, Fellow, IEEE Absrac I his paper, we cosider a recursive esimaio problem for liear regressio where he sigal o be esimaed admis a sparse represeaio ad measureme samples are oly sequeially available. We propose a coverge parallel esimaio scheme ha cosiss of solvig a sequece of l -regularized leassquare problems approximaely. The proposed scheme is ovel i hree aspecs: all elemes of he uow vecor variable are updaed i parallel a each ime isa, ad he covergece speed is much faser ha sae-of-he-ar schemes which updae he elemes sequeially; boh he updae direcio ad sepsize of each eleme have simple closed-form expressios, so he algorihm is suiable for olie real-ime implemeaio; ad 3 he sepsize is desiged o accelerae he covergece bu i does o suffer from he commo iricacy of parameer uig. Boh ceralized ad disribued implemeaio schemes are discussed. The aracive feaures of he proposed algorihm are also illusraed umerically. Idex Terms LASSO, liear regressio, miimizaio sepsize rule, parallel algorihm, recursive esimaio, sparse sigal processig, sochasic opimizaio. I. INTRODUCTION SIGNAL esimaio has bee a fudameal problem i a umber of scearios, such as wireless sesor ewors WSN ad cogiive radio CR. WSN has received a lo of aeio ad is foud applicaio i diverse disciplies such as eviromeal moiorig, smar grids, ad wireless commuicaios ]. CR appears as a eablig echique for flexible Mauscrip received March 7, 05; revised Sepember 5, 05 ad February 4, 06; acceped April 9, 06. Dae of publicaio May 0, 06; dae of curre versio Augus 05, 06. The wor of Y. Yag was pored by he Seveh Framewor Programme for Research of he Europea Commissio uder Gra ADEL-69647, he EXPRESS projec wihi he DFG prioriy program CoSIP DFG-SPP 798, ad he Hog Kog RGC research gra. The wor of M. Pesaveo was pored by he Seveh Framewor Programme for Research of he Europea Commissio uder Gra ADEL ad he EXPRESS projec wihi he DFG prioriy program CoSIP DFG-SPP 798. The wor of M. Zhag ad D. P. Palomar was pored by he Hog Kog RGC research gra. The maerial i his paper was preseed a he Asilomar Coferece o Sigals, Sysems, ad Compuers, Pacific Grove, CA, USA, November 04 ]. The associae edior coordiaig he review of his mauscrip ad approvig i for publicaio was Prof. Cédric Richard. Y. Yag is wih he Iel Deuschlad GmbH, Neubiberg 85579, Germay yag.yag@iel.com. M. Pesaveo is wih he Commuicaio Sysems Group, Darmsad Uiversiy of Techology, Darmsad 6483, Germay pesaveo@. u-darmsad.de. M. Zhag is wih he Deparme of Compuer Sciece ad Egieerig, The Chiese Uiversiy of Hog Kog, Hog Kog zhagmy@cse.cuh.edu.h. D. P. Palomar is wih he Deparme of Elecroic ad Compuer Egieerig, The Hog Kog Uiversiy of Sciece ad Techology, Kowloo, Hog Kog palomar@us.h. Color versios of oe or more of he figures i his paper are available olie a hp://ieeexplore.ieee.org. Digial Objec Ideifier 0.09/TSIPN ad efficie use of he radio specrum 3], 4], sice i allows ulicesed secodary users SUs o access he specrum provided ha he licesed primary users PUs are idle, ad/or he ierferece geeraed by he SUs is below a cerai level ha is olerable for he PUs 5], 6]. Oe prerequisie i CR sysems is he abiliy o obai a precise esimae of he PUs power disribuio map so ha he SUs ca avoid he areas i which he PUs are acively rasmiig. This is usually realized hrough he esimaio of he posiio, rasmi saus, ad/or rasmi power of PUs 7] 0], ad he esimaio is ypically obaied based o he miimum mea-square-error MMSE crierio ], 9], ] 5]. The MMSE approach ivolves he calculaio of he expecaio of a squared l -orm fucio ha depeds o he so-called regressio vecor ad measureme oupu, boh of which are radom variables. This is esseially a sochasic opimizaio problem, bu whe he saisics of hese radom variables are uow, i is impossible o calculae he expecaio aalyically. A aleraive is o use he sample average fucio, cosruced from sequeially available measuremes, as a approximaio of he expecaio, ad his leads o he wellow recursive leas-square RLS algorihm ], ] 4]. As he measuremes are available sequeially, a each ime isa of he RLS algorihm, a LS problem has o be solved, which furhermore admis a closed-form soluio ad hus ca efficiely be compued. More deails ca be foud i sadard exboos such as ], ]. I pracice, he sigal o be esimaed may be sparse i aure ], 8], 9], 5], 6]. I a rece aemp o apply he RLS approach o esimae a sparse sigal, a regularizaio fucio i erms of l -orm was icorporaed io he LS fucio o ecourage sparse esimaes ], 5] 9], leadig o a l - regularized LS problem which has he form of he leas-absolue shriage ad selecio operaor LASSO 0]. The i he recursive esimaio of a sparse sigal, he fudameal differece from he sadard RLS approach is ha a each ime isa, isead of solvig a LS problem as i he RLS algorihm, a l -regularized LS problem i he form of LASSO is solved ]. However, a closed-form soluio o he l -regularized LS problem does o exis because of he l -orm regularizaio fucio ad he problem ca oly be solved ieraively. As a maer of fac, ieraive algorihms o solve he l -regularized LS problems have bee he ceer of exesive research i rece years ad a umber of solvers have bee developed, e.g., GP ], l ls ], FISTA 3], ADMM 4], FLEXA 5], ad DQP-LASSO 6]. Sice he measuremes are sequeially available, ad wih each ew measureme, a ew l -regularized X 06 IEEE. Persoal use is permied, bu republicaio/redisribuio requires IEEE permissio. See hp:// sadards/publicaios/righs/idex.hml for more iformaio.

2 YANG e al.: ONLINE PARALLEL ALGORITHM FOR RECURSIVE ESTIMATION OF SPARSE SIGNALS 9 LS problem is formed ad solved, he overall complexiy of usig solvers for he whole sequece of l -regularized LS problems is o loger affordable. If he evirome is furhermore rapidly chagig, his mehod is o suiable for real-ime applicaios as ew measuremes may already arrive before he previous l -regularized LS problem is solved. To reduce he complexiy of he esimaio scheme so ha i is suiable for olie real-ime implemeaio, he auhors i 5], 7], 8] proposed algorihms i which he l -regularized LS problem a each ime isa is solved oly approximaely. For example, i he algorihm proposed i 5], a each ime isa, he l -regularized LS problem is solved wih respec o w.r.. oly a sigle eleme of he uow vecor variable while he remaiig elemes are fixed, ad he updae of ha eleme has a simple closed-form expressio based o he socalled sof-hresholdig operaor 3]. Wih he ex measureme ha arrives, a ew l -regularized LS problem is formed ad solved w.r.. he ex eleme oly while he remaiig elemes are fixed. This sequeial updae rule is ow i lieraure as he bloc coordiae desce mehod 7]. Iuiively, sice oly a sigle eleme is updaed a each ime isa, he olie sequeial algorihm proposed i 5] someimes suffers from slow covergece, especially whe he sigal has a large dimesio while large dimesios of sparse sigals are uiversal i pracice. I is empig o use a parallel scheme i which he updae direcios of all elemes are compued ad updaed simulaeously a each ime isa, bu he covergece properies of parallel algorihms are mosly ivesigaed for deermiisic opimizaio problems see 5] ad he refereces herei ad hey may o coverge for he sochasic opimizaio problem a had. Besides his, he covergece speed of parallel algorihms heavily depeds o he choice of he sepsizes. Typical rules for choosig he sepsizes are he Armijo-lie successive lie search rule, cosa sepsize rule, ad dimiishig sepsize rule. The former wo suffer from high complexiy ad slow covergece 5, Remar 4], while he decay rae of he dimiishig sepsize is very difficul o choose: o he oe had, a slowly decayig sepsize is preferable o mae oable progress ad o achieve saisfacory covergece speed; o he oher had, heoreical covergece is guaraeed oly whe he sepsizes decays fas eough. I is a difficul as o is ow o fid he decay rae ha yields a good rade-off. Sparsiy-aware learig over ewor algorihms have bee proposed i 6], 8] 30]. They are suiable for disribued implemeaio, bu hey do o coverge o he exac MMSE esimae. Oher schemes suiable for he olie esimaio of sparse sigals iclude LMS-ype algorihms 8], 3] 33]. However, heir covergece speed is ypically slow ad he free parameers e.g., sepsizes are difficul o choose: eiher he selecio of he free parameers depeds o iformaio ha is o easily obaiable i pracice, such as he saisics of he regressio vecor, or he covergece is very sesiive o he choice of he free parameers. A rece wor o parallel algorihms for sochasic opimizaio is 34]. However, he algorihms proposed i 34] are o applicable for he recursive esimaio of sparse sigals. This is because he regularizaio fucio i 34] mus be srogly covex ad differeiable while he regularizaio gai mus be lower bouded by some posiive cosa so ha covergece ca be achieved. However he regularizaio fucio i erms of l -orm for sparse sigal esimaio is covex bu o srogly covex ad odiffereiable while he regularizaio gai is decreasig o zero. I his paper, we propose a olie parallel algorihm wih provable covergece for recursive esimaio of sparse sigals. I paricular, our mai coribuios are summarized as follows. Firsly, a each ime isa, he l -regularized LS problem is solved approximaely ad all elemes are updaed i parallel, so he covergece speed is grealy ehaced compared wih 5]. As a orivial exesio of 5] from sequeial updae o parallel updae, ad of 5], 35] from deermiisic opimizaio problems o sochasic opimizaio problems, he covergece of he proposed algorihm is esablished. Secodly, we propose a ew procedure for he compuaio of he sepsize based o he so-called miimizaio rule also ow as exac lie search ad is beefis are wofold: firsly, i is esseial for he covergece of he proposed algorihm, which may however diverge uder oher sepsize rules; secodly, oable progress is achieved afer each variable updae ad he commo iricacy of complicaed parameer uig is saved. Besides his, boh he updae direcio ad sepsize of each eleme exhibi simple closed-form expressios, so he proposed algorihm is fas o coverge ad suiable for olie implemeaio. The res of he paper is orgaized as follows. I Secio II we iroduce he sysem model ad formulae he recursive esimaio problem. The olie parallel algorihm is proposed i Secio III, ad is implemeaios ad exesios are discussed i Secio IV. The performace of he proposed algorihm is evaluaed umerically i Secio V ad fially cocludig remars are draw i Secio VI. Noaio: We use x, x ad X o deoe scalar, vecor ad marix, respecively. X j is he j, h eleme of X; x ad x j, is he h eleme of x ad x j, respecively, ad x =x K = ad x j =x j, K =.Weusex o deoe he elemes of x excep x : x x j K j=,j. We deoe dx as a vecor ha cosiss of he diagoal elemes of X, diagx as a diagoal marix whose diagoal elemes are he same as hose of X, ad diagx as a diagoal marix whose diagoal vecor is x, i.e., diagx =diagdx. The operaor x] b a deoes he elemewise projecio of x oo a, b]: x] b a maxmix, b, a, ad x] + deoes he eleme-wise projecio of x oo he oegaive orha: x] + maxx, 0. The Moore Perose iverse of X is deoed as X, ad λ max X deoes he larges eigevalue of X. II. SYSTEM MODEL AND PROBLEM FORMULATION Suppose x =x K = RK is a deermiisic sparse sigal o be esimaed based o he he measureme y R, ad boh quaiies are coeced hrough a liear regressio model: y = g T x + v, =,...,N,

3 9 IEEE TRANSACTIONS ON SIGNAL AND INFORMATION PROCESSING OVER NETWORKS, VOL., NO. 3, SEPTEMBER 06 where N is he umber of measuremes a ay ime isa. The regressio vecor g =g, K = RK is assumed o be ow, ad v R is he addiive esimaio oise. Throughou he paper, we mae he followig assumpios o g ad v for =,...,N: A. g is a radom variable wih a bouded posiive defiie covariace marix; A. v is a radom variable wih zero mea ad bouded variace; A.3 g ad v are ucorrelaed. Someimes we may also eed bouded assumpios o he higher order momes of g ad v : A. g is a radom variable whose covariace marix is posiive defiie ad whose momes are bouded; A. v is a radom variable wih zero mea ad bouded momes; A.3 g ad v are ucorrelaed. Give he liear model i, he problem is o esimae x from he se of regressio vecors ad measuremes {g,y } N =. Sice boh he regressio vecor g ad esimaio oise v are radom variables, he measureme y is also radom. A fudameal approach o esimae x is based o he MMSE crierio, which has a solid roo i adapive filer heory ], ]. To improve he esimaio precisio, all available measuremes {g,y } N = are exploied o form a cooperaive esimaio problem which cosiss of fidig he variable ha miimizes he mea-square-error ], 9], 36]: N x = arg mi E y g T x ] x=x K = = = arg mi x xt Gx b T x, where G N = E ] g g T ad b N = E y g ], ad he expecaio is ae over {g,y } N =. I pracice, he saisics of {g,y } N = are ofe o available o compue G ad b aalyically. I fac, he absece of saisical iformaio is a geeral rule raher ha a excepio. A commo approach is o approximae he expecaio i by he sample average fucio cosruced from he measuremes or realizaios {g τ,y τ } τ = sequeially available up o ime ]: x rls arg mi x xt G x b T x 3a = G b, 3b where G ad b is he sample average of G ad b, respecively: G g τ g τ T, b y τ g τ. τ = = τ = = 4 I lieraure, 3 is ow as RLS, as idicaed by he subscrip rls, ad x rls ca be compued efficiely i closedform, cf. 3b. Noe ha i 4 here are N measuremes y τ, g τ N = available a each ime isa τ. For example, i a WSN, y τ, g τ is he measureme available a he sesor. I may pracical applicaios, he uow sigal x is sparse by aure or by desig, bu x rls give by 3 is o ecessarily sparse whe is small 0], ]. To overcome his shorcomig, a sparsiy ecouragig fucio i erms of l - orm is icorporaed io he sample average fucio i 3, leadig o he followig l -regularized sample average fucio a ay ime isa =,,...], 8], 5]: L x xt G x b T x + μ x, 5 where μ > 0. Defie x L x: as he miimizig variable of x = arg mi L x, =,,..., 6 x I lieraure, problem 6 for ay fixed is ow as he leasabsolue shriage ad selecio operaor LASSO 0], ] as idicaed by he subscrip i 6. Noe ha i bach processig 0], ], problem 6 is solved oly oce whe a cerai umber of measuremes are colleced so is equal o he umber of measuremes, while i he recursive esimaio of x, he measuremes are sequeially available so is icreasig ad 6 is solved repeaedly a each ime isa =,,... The advaage of 6 over, whose objecive fucio is sochasic ad whose calculaio depeds o uow parameers G ad b, is ha 6 is a sequece of deermiisic opimizaio problems whose heoreical ad algorihmic properies have bee exesively ivesigaed ad widely udersood. A aural quesio arises i his coex: is 6 equivale o i he sese ha x is a srogly cosise esimaor of x, i.e., lim x = x wih probabiliy oe? The relaio bewee x i 6 ad he uow variable x is give i he followig lemma 5]. Lemma : Suppose Assumpio A as well as he followig assumpios are saisfied for problem 6: A g y, respecively is a idepede ideically disribued i.i.d. radom process wih he same probabiliy desiy fucio of g y, respecively. A3 { μ } is a posiive sequece covergig o 0, i.e., μ > 0 ad lim μ =0. The lim x = x wih probabiliy oe. A example of μ saisfyig Assumpio A3 is μ = α/ β wih α>0 ad β>0. Typical choices of β are β = ad β =0.5 5]. Noe ha he dimiishig regularizaio gai μ differeiaes our wor from 37] i which he sparsiy regularizaio gai is a posiive cosa μ = μ for some μ>0: he algorihms proposed i 37] does o ecessarily coverge o x while he algorihm o be proposed i he ex secio does. Lemma o oly saes he relaio bewee x ad x from a heoreical perspecive, bu also suggess a simple algorihmic soluio for problem : x ca be esimaed by solvig a sequece of deermiisic opimizaio problems 6, oe for

4 YANG e al.: ONLINE PARALLEL ALGORITHM FOR RECURSIVE ESTIMATION OF SPARSE SIGNALS 93 each ime isa =,,... However, i coras o he RLS algorihm i which each updae has a closed-form expressio, cf. 3b, problem 6 does o have a closed-form soluio ad i ca oly be solved umerically by a ieraive algorihm such as GP ], l ls ], FISTA 3], ADMM 4], ad FLEXA 5]. As a resul, solvig 6 repeaedly a each ime isa =,,... is eiher compuaioally pracical or real-ime applicable. The aim of he followig secios is o develop a algorihm ha ejoys easy implemeaio ad fas covergece. III. THE PROPOSED ONLINE PARALLEL ALGORITHM The LASSO problem i 6 is covex, bu he objecive fucio is odiffereiable ad i cao be miimized i closedform, so solvig 6 compleely w.r.. all elemes of x by a solver a each ime isa =,,... is eiher compuaioally pracical or suiable for olie implemeaio. To reduce he complexiy of he variable updae, a algorihm based o iexac opimizaio is proposed i 5]: a ime isa, oly a sigle eleme x wih = mod,k+is updaed by is so-called bes respose, i.e., L x is miimized w.r.. x oly: x + = arg mi L x, x wih x x j j, which ca be solved i closed-form, while he remaiig elemes {x j } j remai uchaged, i.e., x + = x. A he ex ime isa +, a ew sample average fucio L + x is formed wih ewly arrivig measuremes, ad he +h eleme, x +, is updaed by miimizig L + x w.r.. x + oly, while he remaiig elemes agai are fixed. Alhough easy o impleme, sequeial updaig schemes updae oly a sigle eleme a each ime isa ad hey someimes suffer from slow covergece whe he umber of elemes K is large. To overcome he slow covergece of he sequeial updae, we propose a olie parallel updae scheme, wih provable covergece, i which 6 is solved approximaely by simulaeously updaig all elemes oly oce based o heir idividual bes respose. Give he curre esimae x which is available before he h measureme arrives, he esimae updae x + is deermied based o all he measuremes colleced up o ime isa i a hree-sep procedure as described ex. Sep Updae Direcio: I his sep, all elemes of x are updaed i parallel ad he updae direcio of x a x = x, deoed as ˆx x, is deermied based o he bes-respose ˆx. For each eleme of x,sayx, is bes respose a x = x is give by: ˆx { arg mi L x, x x + c x x }, 7 where x {x j } j ad i is fixed o he values of he precedig ime isa x = x. A addiioal quadraic proximal erm wih c > 0 is icluded i 7 for umerical simpliciy ad sabiliy 7], because i plays a impora role i he covergece aalysis of he proposed algorihm; cocepually i is a pealy wih variable weigh c for movig away from he curre esimae x. x could be arbirarily chose, e.g., x = 0. Afer subsiuig 5 io 7, he bes-respose i 7 ca be expressed i closed-form: ˆx = arg mi x = S μ r S μ G x r x + μ x + c x x x +c x G + c or compacly: ˆx =ˆx K = ad ˆx = diag G + diag c r x + diag where ad r x = r x K diag = G x S a b b a] + b a] +, =,...,K, 8 c x, 9 G x b, 0 is he well-ow sof-hresholdig operaor 3], 38]. From he defiiio of G i 4, G 0 ad G 0 for all, so he marix iverse i 9 is defied. Give he updae direcio ˆx x, a iermediae updae vecor x γ is defied x γ =x + γ ˆx x, where γ 0, ] is he sepsize. The updae direcio ˆx x is a desce direcio of L x i he sese specified by he followig proposiio. Proposiio Desce Direcio: For ˆx =ˆx K = give i 9 ad he updae direcio ˆx x, he followig holds for ay γ 0, ]: x L γ L x γ c mi λ max G γ ˆx x, where c mi mi { G + } c > 0. Proof: The proof follows he same lie of aalysis i 5, Prop. 8c] ad is hus omied here. Sep Sepsize: I his sep, he sepsize γ i is deermied so ha fas covergece is observed. I is easy o see from ha for sufficiely small γ, he righ had side of becomes egaive ad L x γ decreases as compared o L x. Thus, o miimize L x γ, a aural choice of he sepsize rule is he so-called miimizaio rule 39, Sec...] also ow as he exac lie search 40, Due o he diagoal srucure of diagg +diagc, he marix iverse ca be compued from he scalar iverse of he diagoal elemes

5 94 IEEE TRANSACTIONS ON SIGNAL AND INFORMATION PROCESSING OVER NETWORKS, VOL., NO. 3, SEPTEMBER 06 Sec. 9.], which is he sepsize, deoed as γ op, ha decreases L x γ o he larges exe: γ { op =argmi L x γ L x } 0 γ ˆx x T G ˆx x γ =argmi 0 γ +G x b T ˆx x. γ +μ x + γˆx x x 3 Therefore by defiiio of γ op we have for ay γ 0, ]: L x + γ op ˆx x L x + γ ˆx x. 4 However, he applicabiliy of he sadard miimizaio rule 3 is usually limied i pracice because of he high compuaioal complexiy of solvig he opimizaio problem i 3. I paricular, he odiffereiable l -orm fucio maes i impossible o fid a closed-form expressio of γ op ad he problem i 3 ca oly be solved umerically by a solver such as SeDuMi 4]. To obai a sepsize ha exhibis a good rade-off bewee covergece speed ad compuaioal complexiy, we propose a simplified miimizaio rule which yields fas covergece bu ca be compued a a low complexiy. Firsly, oe ha he high complexiy of he sadard miimizaio rule lies i he odiffereiable l -orm fucio i 3. I follows from he covexiy of orm fucios ha for ay γ 0, ]: μ x + γˆx x x = μ γx + γˆx μ x γμ x + γμ ˆx μ x 5a = μ ˆx x γ. 5b The righ had side of 5b is liear i γ, ad equaliy is achieved i 5a eiher whe γ =0or γ =. I he proposed simplified miimizaio rule, isead of direcly miimizig L x γ L x over γ, is upper boud based o 5 is miimized: ˆx x T ˆx G x γ γ arg mi + G x b T ˆx x γ. 0 γ + μ ˆx x γ 6 The scalar opimizaio problem i 6 cosiss of a covex quadraic objecive fucio alog wih a simple boud cosrai ad i has a closed-form soluio, give by 7 a he boom of his page. I is easy o verify ha γ is obaied by projecig he ucosraied opimal variable of he covex quadraic problem i 6 oo he ierval 0, ]. The advaage of miimizig he upper boud fucio of L x γ i 6 is ha he opimal γ, deoed as γ, always has a closed-form expressio, cf. 7. A he same ime, i also yields a decrease i L x a x = x as he sadard miimizaio rule γ op 3 does i 4, ad his decreasig propery is saed i he followig proposiio. Proposiio 3: Give x γ ad γ defied i ad 6, respecively, he followig holds: L x γ L x, ad equaliy is achieved if ad oly if γ =0. Proof: Deoe he objecive fucio i 6 as L x γ. I follows from 5 ha L x γ L x L x γ, 8 ad equaliy i 8 is achieved whe γ =0ad γ =. Besides his, i follows from he defiiio of γ ha L x γ L x γ γ =0 = L x. 9 Sice he opimizaio problem i 6 has a uique opimal soluio γ give by 7, equaliy i 9 is achieved if ad oly if γ =0. Fially, combiig 8 ad 9 yields he coclusio saed i he proposiio. The sigalig required o perform 7 ad also 9 whe implemeed disribuedly will be discussed i Secio IV. Sep 3 Dyamic Rese: I his sep, he esimae updae x + is defied based o x γ give i ad 7. We firs remar ha alhough x γ yields a lower value of L x ha x, i is o ecessarily he soluio of he opimizaio problem i 6, i.e., L x L x γ L x =mil x. x 0 This is because x is updaed oly oce from x = x o x = x γ, which i geeral ca be furher improved uless x γ =x, i.e., x γ already miimizes L x. The defiiios of L x ad x 0=L x L x x=0 i 5-6 reveal ha,=,,... γ = G x b T ˆx x +μ ˆx x ˆx x T G ˆx x ] 0. 7

6 YANG e al.: ONLINE PARALLEL ALGORITHM FOR RECURSIVE ESTIMATION OF SPARSE SIGNALS 95 Algorihm : The Olie Parallel Algorihm for Recursive Esimaio of Sparse Sigals. Iiializaio: x = 0, =. A each ime isa =,,...: Sep : Calculae ˆx accordig o 9. Sep : Calculae γ accordig o 7. Sep 3-: Calculae x γ accordig o. Sep 3-: Updae x + accordig o 3. Depedig o wheher L x is smaller ha 0 or o, i is possible o relae 0 ad i he followig hree ways: 0=L 0 L x L x γ L x, L x 0=L 0 L x γ L x, L x L x γ 0=L 0 L x. The las case i implies ha x γ is o ecessarily beer ha he poi 0. Therefore we defie he esimae updae x + o be he bes poi bewee he wo pois x γ ad 0: x + = arg mi L x x { x +,0} x γ, if L x γ L 0 =0, = 0, oherwise, 3 ad i is sraighforward o ifer he followig relaioship amog x, x γ, x + ad x : L x L x γ L x + L x. Moreover, he dyamic rese 3 guaraees ha x + { x : L x 0 },=,,... 4 Sice lim G 0 ad b coverges from Assumpios A-A, 4 guaraees ha { x } is a bouded sequece. Remar 4: Alhough L x + 0 for ay accordig o 4, i may happe ha L + x + > 0 uless x + = 0, which correspods o he firs wo cases i. The las case i is hus sill possible ad i is ecessary o chec if L + x + γ + 0 as i 3. To summarize he above developme, he proposed olie parallel algorihm is formally described i Algorihm. To aalyze he covergece of Algorihm, we assume ha he sequece {μ } moooically decreases o 0: A3 { μ } is a posiive decreasig sequece covergig o 0, i.e., μ + μ > 0 for all ad lim μ = 0. We also assume ha c is seleced such ha: A4 G + c c for some c>0ad all =,...,K. Theorem 5 Srog Cosisecy: Suppose Assumpios A, A, A3 ad A4 are saisfied. The x is a srogly cosise esimaor of x, i.e., lim x = x wih probabiliy oe. Proof: See Appedix A. Assumpio A is sadard o radom variables ad is usually saisfied i pracice. We ca see from Assumpio A4 ha if here already exiss some value c>0such ha G c for all, he quadraic proximal erm i 7 is o loger eeded, i.e., we ca se c =0wihou affecig covergece. This is he case whe is sufficiely large because lim G 0. I pracice i may be difficul o decide if is large eough, so we ca jus assig a small value o c for all i order o guaraee he covergece. As for Assumpio A3, i is saisfied by he previously meioed choices of μ, e.g., μ = α/ β wih α>0ad 0.5 β. Theorem 5 esablishes ha here is o loss of srog cosisecy if a each ime isa, 6 is solved oly approximaely by updaig all elemes simulaeously based o he bes-respose oly oce. I wha follows, we comme o some of he desirable feaures of Algorihm ha mae i appealig i pracice: Algorihm belogs o he class of parallel algorihms where all elemes are updaed simulaeously a each ime isa. Compared wih sequeial algorihms where oly oe eleme is updaed a each ime isa 5], he improveme i covergece speed is oable, especially whe he sigal dimesio is large. This is illusraed umerically i Secio V cf. Figs. ad. Algorihm is easy o impleme ad suiable for olie implemeaio, sice boh he compuaios of he besrespose ad he sepsize have closed-form expressios. Wih he simplified miimizaio sepsize rule, a oable decrease i objecive fucio value is achieved afer each variable updae, ad he difficuly of uig he decay rae of he dimiishig sepsize as required i 35] is saved. Mos imporaly, he algorihm may o coverge uder decreasig sepsizes. 3 Algorihm coverges uder milder assumpios ha sae-of-he-ar algorihms. The regressio vecor g ad he oise v do o eed o be uiformly bouded, which is required i 4], 43] ad which is o saisfied i case of ubouded disribuios, e.g., i he Gaussia disribuio. IV. IMPLEMENTATION AND EXTENSIONS A. A Special Case: x 0 The proposed Algorihm ca be furher simplified if x,he sigal o be esimaed, has addiioal properies. For example, i he coex of CR sudied i 8], x represes he power vecor ad i is by defiiio always oegaive. I his case, a oegaive cosrai o x i 7 is eeded: { L x, x ˆx = arg mi x 0 ˆx = + c ad he bes-respose ˆx i 9 simplifies o ] + r + c x μ G + c x x },,=,...,K.

7 96 IEEE TRANSACTIONS ON SIGNAL AND INFORMATION PROCESSING OVER NETWORKS, VOL., NO. 3, SEPTEMBER 06 Furhermore, sice boh x ad ˆx are oegaive, we have x + γˆx x 0, 0 γ, ad K x + γˆx x = = x = K = x + γˆx x + γ ˆx x ]. Therefore he sadardmiimizaio rule 3 ca be adoped direcly ad he sepsize is accordigly give as γ = G x b + μ T ˆx x ], ˆx x T G ˆx x 0 where is a vecor wih all elemes equal o. B. Implemeaio Deails ad Complexiy Aalysis Algorihm ca be implemeed i a ceralized ad parallel or a disribued ewor archiecure. To ease he exposiio, we discuss he implemeaio deails i he coex of a WSN wih a oal umber of N odes. Newor Wih a Fusio Ceer: The fusio ceer firs performs he compuaio of 9 ad 7. Towards his ed, sigalig from he sesors o he fusio ceer is required: a each ime isa, each sesor seds he values g,y R K + o he fusio ceer. Noe ha G ad b defied i 4 ca be updaed recursively G = G + b = b + = = g g T, 5a y g. 5b The afer updaig x accordig o ad 3, he fusio ceer seds x + R K bac o all sesors. We ex discuss he compuaioal complexiy of Algorihm. Noe ha i 5, he ormalizaio by is immaerial as i appears i boh he umeraor ad deomiaor. Amog ohers, N + K + K/ muliplicaios ad addiios are required o compue 5a. Besides his, 3K muliplicaios ad 3KK addiios are required o perform he marix-vecor muliplicaios G x of 0, G ˆx x of 4 ad G x γ of 3. I is possible o verify ha hese operaios domiae he ohers i erms of muliplicaios ad addiios, ad he overall compuaioal complexiy is he same as he radiioal RLS algorihm, Ch. 4]. We furher remar ha he compuaios specified i 9, 7 ad 3, e.g., he marix-vecor ad eleme-wise vecorvecor muliplicaios, are easily parallelizable by usig parallel hardware e.g., FPGA or muliple processors/cores. I his case, he compuaio ime could be sigificaly reduced ad his is of grea ieres i a ceralized ewor as well. Newor Wihou a Fusio Ceer: I his case, he compuaioal ass are evely disribued amog he sesors ad he compuaio i each sep of Algorihm is performed locally by each sesor a he price of some sigalig exchage amog differe sesors. We firs defie he sesor-specific variables G sesor as: G g τ g τ τ = T, ad b = τ = ad b for y g, 3 so ha G = N = G ad b = N = b. Noe ha G ad b ca be compued locally by sesor wihou ay sigalig exchage required. I is also easy o verify ha, similar o 5, G ad b ca be updaed recursively by sesor, so he sesors do o have o sore all pas daa. The iformaio exchage amog sesors i carried ou i wo phases. Firsly, for sesor, o perform 9 Sep of Algorihm ], dg ad r are required, 33 ad hey ca be decomposed as follows: dg = G x b = = = dg R K, 7a G x b R K. 7b Furhermore, o deermie he sepsize 7 Sep of Algorihm ], he followig compuaios mus be available a sesor : ad G x = G ˆx = = = G x R K 7c G ˆx R K, 7d G x b T ˆx x N T = G x b ˆx x, 7e = however, compuig 7e does o require ay addiioal sigalig sice N = G x b is already available from 7b. Wih ˆx ad γ, each sesor ca locally calculae x γ accordig o Sep 3- of Algorihm ]. Noe ha L x γ Sep 3- of Algorihm ] ca be compued based o available iformaio 7b 7d because L x γ = x γ T G x γ b T x γ +μ x γ 3 Recall ha diagg =diagdg.

8 YANG e al.: ONLINE PARALLEL ALGORITHM FOR RECURSIVE ESTIMATION OF SPARSE SIGNALS 97 = x γ T G x γ b + μ x γ = x γ T G x b G x + γ G ˆx x + μ x γ, where G x b comes from 7b, G x comes from 7c, ad G ˆx x comes from 7c ad 7d. We ca also ifer from he above discussio ha he mos complex operaios a each ode are he compuaio of G i 6, which cosiss of K + K/ muliplicaios ad addiios, ad he marix-vecor muliplicaios G x i 7b, 7c ad G ˆx i 7d, each of which cosiss of K muliplicaios ad KK addiios, leadig o a oal of.5k +0.5K muliplicaios ad KK addiios. To summarize, i he firs phase, each ode eeds o exchage dg, G x b R K, while i he secod phase, he sesors eed o exchage G x, G ˆx R K ; hus he dimesio of he vecor ha eeds o be exchaged a each ime isa is 4K. Iwhafollows,wedraw several commes o he iformaio exchage ad is implicaios. The dimesio of he vecor o be exchaged is much smaller ha i ] ad 8]. For example i, A.5], he opimizaio problem 6 is solved exacly a each ime isa whereas i is solved oly approximaely i he proposed Algorihm, cf. 0. I his sese i is esseially a double layer algorihm: i he ier layer, a ieraive algorihm is used o solve 6 while i he ouer layer is icreased o +ad 6 is solved agai. Suppose he ieraive algorihm i he ier layer coverges i T ieraios; i geeral T. I each ieraio of he ier layer, he sesors should exchage a vecor of he size K, ad his is repeaed uil he ermiaio of he ier layer, leadig o a oal size of T K, which is much larger ha ha of he proposed algorihm, amely, 4K. Furhermore, sice he iformaio exchage mus be repeaed for T imes a each ime isa, he icurred laecy is much loger ha ha of he proposed algorihm, i which he iformaio exchage is carried ou oly wice. The aalysis for he disribued implemeaio of 5], proposed i 8], is similar ad hus omied. I pracice, he iformaio exchage could be realized by broadcas, or cosesus algorihms if oly local commuicaio wih eighbor odes is possible. Sice cosesus algorihms are of a ieraive aure, he proposed disribued algorihm would have a addiioal ier layer if he cosesus algorihm were explicily coued: i he ouer layer, he sesors perform he esimae updae ad 3; i he ier layer, he sesors compue he average values 7 usig a ieraive cosesus algorihm The wo-layer srucure of he proposed algorihm is differe from ha of ], 8]: sice he average values i ], 8] are also compued usig a ieraive cosesus algorihm, he algorihms proposed i ], 8] would have hree layers if he cosesus algorihm were explicily coued. Sice he covergece of Algorihm is based o perfec iformaio exchage, we should use cosesus algorihms uder which he exac cosesus is reached i a fiie umber of seps, for example, 44]. More specifically, he exac cosesus i 44] is achieved i a mos T max N + mi N seps, where N is he umber of eighbors of he sesor,so he oal sigalig overhead a each ime isa of Algorihm is4t max K. However, his specific choice of cosesus algorihm imposes addiioal cosrais o he ewor ad he sesors for example, each sesor should have he owledge of opology of he global ewor ad addiioal coordiaio is required amog he sesors, which may impair he applicabiliy of he proposed algorihm. If cosesus algorihms wih asympoic covergece are used for iformaio exchage, hey are ypically ermiaed afer fiie ieraios i pracice. The he iformaio available a each sesor is a oisy esimae of he real iformaio ad he proposed algorihm may o coverge. The covergece i his case requires furher ivesigaio. C. Time- ad Norm-Weighed Sparsiy Regularizaio For a give vecor x, is por S x is defied as he se of idices of ozero elemes: S x { K : x 0}. Suppose wihou loss of geeraliy ha S x = {,,..., x 0 }, where x 0 is he umber of ozero elemes of x. I is show i 5] ha wih he ime-weighed sparsiy regularizaio 6, he esimae x does o ecessarily saisfy he so-called oracle properies : a esimaor x is said o saisfy he oracle properies if lim Prob S x = S x ]=, 8a ad x : x x 0 : x 0 d N 0,σ G : x 0,: x 0, 8b where d meas covergece i disribuio ad G :,: is he upper lef bloc of G. The firs propery 8a ad he secod propery 8b is called por cosisecy ad -esimaio cosisecy, respecively 5]. To mae he esimaio saisfy he oracle properies, i was suggesed i 5] ha a ime- ad orm-weighed LASSO ca be used, ad he loss fucio L x i 5 ca be modified as follows: R L x = τ = = y τ + μ K = g τ T x W μ x rls, x, 9 where x rls is give i 3; lim μ =0 ad lim μ =,soμ mus decrease slower ha / ;

9 98 IEEE TRANSACTIONS ON SIGNAL AND INFORMATION PROCESSING OVER NETWORKS, VOL., NO. 3, SEPTEMBER 06 3 he weigh facor W μ x is defied as, if x μ, aμ x W μ x, if μ x aμ, a μ 0, if x aμ, ad a>is a give cosa. Therefore, he value of he weigh fucio μ W μ x rls, i 9 depeds o he relaive magiudes of μ ad x rls,. Afer replacig he uiversal sparsiy regularizaio gai μ by μ W μ x rls, for each eleme x i 9 ad 7, Algorihm ca readily be applied o esimae x based o he ime- ad orm-weighed loss fucio 9 ad he srog cosisecy also holds. To see his, we oly eed o verify he oicreasig propery of he weigh fucio μ W μ x rls,. We remar ha whe is sufficiely large, i is eiher μ W μ x rls, =0or μ W μ x rls, =μ. This is because lim x rls = x uder he codiios of Lemma. If x > 0, sice lim μ =0, here exiss for ay arbirarily small ɛ>0 some 0 such ha aμ <x ɛ for all 0; he weigh facor i his case is 0 for all 0, ad he oicreasig propery is auomaically saisfied. If, o he oher had, x =0, he x rls coverges o x =0a a rae of / 45]. Sice μ decreases slower ha /, here exiss some 0 such ha x rls, <μ for all 0. I his case, W μ x rls, is equal o ad he weigh facor is simply μ for all 0, which is oicreasig. D. Recursive Esimaio of Time-Varyig Sigals If he sigal o be esimaed is ime-varyig, he loss fucio 5 eeds o be modified i a way such ha he ew measureme samples are give more weigh ha he old oes. Defiig he so-called forgeig facor β, where 0 <β<, he ew loss fucio is give as ], ], 5]: mi x = τ = β τ g τ T x y τ + μ x. 30 We observe ha whe β =, 30 is as same as 5. I his case, he oly modificaio o Algorihm is ha G ad b are updaed accordig o he followig recursive rule: G = βg + g g T, = b = βb + y g. = For problem 30, sice he sigal o be esimaed is imevaryig, he covergece aalysis i Theorem 5 does o hold ay more. However, simulaio resuls show ha here is lile loss of opimaliy whe opimizig 30 oly approximaely by Algorihm. This esablishes he erioriy of he proposed algorihm over he disribued algorihm i ] which solves 30 exacly a he price of a large delay ad a large sigalig burde. Besides his, despie he lac of heoreical aalysis, Algorihm performs beer ha he olie sequeial algorihm 5] umerically, cf. Secio V. V. NUMERICAL RESULTS I his secio, he desirable feaures of he proposed algorihm are illusraed umerically. Uless oherwise saed, he simulaio seup is as follows: he umber of sesors N =, so he subscrip i g is omied; he dimesio of x : K = 00; 3 he proporio of he ozero elemes of x :0.; 4 boh g ad v are geeraed by i.i.d. sadard ormal disribuios: g CN0, I ad v CN0, 0.; 5 he sparsiy regularizaio gai μ =0/; 6 he simulaios resuls are averaged over 00 realizaios. A. Covergece o he Opimal Value We plo i Fig. he relaive error of he objecive value L x L x /L x versus he ime isa for wo dimesios of x wih x 0 = 0, amely, K = 00 i Fig. a ad K = 500 i Fig. b, where x is defied i 6 ad calculaed by MOSEK 46]; x is reured by Algorihm i he proposed olie parallel algorihm coied as parallel algorihm ; 3 x is reured by 5, Algorihm ] i he olie sequeial algorihm coied as sequeial algorihm, where oly oe eleme of x is updaed a each ime isa; 4 i he ehaced sequeial algorihm, all elemes of x are sequeially updaed oce a each ime isa. Defie z, ˆx,...,ˆx,x +,...,x K ]T, K, where ˆx =G + c S μ r x, +c x ; he variable updae i he ehaced sequeial algorihm ca mahemaically be expressed as 55 x + = z,k. 3 Noe ha L x is by defiiio he lower boud of L x ad L x L x 0 for all. From Fig. i is clear ha he proposed algorihm blac curve coverges o a precisio of 0 wih less ha 00 measuremes while he sequeial algorihm blue curve eiher requires may more measuremes cf. Fig. a or does o eve coverge wih a reasoable umber of measuremes cf. Fig. b. The improveme i covergece speed is hus oable, ad he proposed olie parallel algorihm ouperforms he sequeial algorihm boh i covergece speed ad soluio qualiy. Besides his, a compariso of he proposed algorihm for differe sigal dimesios i Fig. a ad b idicaes ha he proposed algorihm scales well ad i is very pracical. We remar ha he compuaioal complexiy per ime isa of he sequeial algorihm 5] is approximaely /K ha of he proposed algorihm, because he former updaes a sigle eleme of x oly accordig o 8, while he laer updaes all 5 The ehaced sequeial algorihm is suggesed by he reviewers.

10 YANG e al.: ONLINE PARALLEL ALGORITHM FOR RECURSIVE ESTIMATION OF SPARSE SIGNALS 99 Fig.. Covergece behavior i erms of objecive fucio value. a Sigal dimesio: K = 00. b Sigal dimesio: K = 500. elemes of x simulaeously based o 9. The compuaioal complexiy per ime isa of he ehaced sequeial algorihm is roughly he same as ha of he proposed algorihm, because he operaio 8 is performed for K imes afer a complee cycle of eleme updaes. However, he associaed compuaioal ime per ime isa of he ehaced sequeial algorihm is a leas K imes as log as ha of he proposed algorihm, because he proposed updae 9 is parallelizable by usig parallel hardware e.g., FPGA or muliple processors/cores. Thus he proposed algorihm is more suiable for olie applicaios. Whe implemeed i a disribued maer, he ehaced sequeial algorihm icurs a large sigalig overhead. Followig he lie of aalysis i Secio IV, we remar ha he odes eed x, b R K afer x, is obaied so ha x,+ ca be compued by each ode locally. A each ime isa, a complee cycle wih K sequeial eleme updaes he leads o a oal dimesio of K, which is much larger ha ha of he proposed algorihm, amely, 4K. The larger sigalig overhead also icreases he laecy. Thus he proposed algorihm is more suiable for disribued implemeaio. Alhough we observe from Fig. ha he proposed algorihm coverges slighly slower ha he ehaced sequeial algorihm, he sigificaly reduced compuaioal ime ad sigalig overhead jusify he erioriy of he proposed algorihm. We also evaluae i Fig. he performace loss icurred by he simplified miimizaio rule 6 idicaed by he blac curve compared wih he sadard miimizaio rule 3 idicaed by he red curve. I is easy o see from Fig. ha hese wo curves almos coicide wih each oher, so he exe o which he simplified miimizaio rule decreases he objecive fucio is early he same as he sadard miimizaio rule ad he performace loss is egligible. o exchage G B. Covergece o he Opimal Variable The we cosider i Fig. he relaive square error x x / x versus he ime isa. To compare he esimaio approaches wih ad wihou sparsiy regularizaio, he RLS algorihm i 3 is also implemeed, where a l regularizaio erm 0 4 x is icluded io 3. Some observaios are i order. We see ha he proposed olie parallel algorihm idicaed by he blac curve ad he ehaced sequeial algorihm idicaed by he red curve wih upper riagular exhibi faser covergece ha oher algorihms. From Fig. we see ha whe he sigal dimesio is icreased from K = 00 o K = 500, he covergece speed of he proposed olie parallel algorihm is o severely slowed dow, which shows ha he proposed algorihm scales well. The ehaced sequeial algorihm coverges slighly faser ha he proposed olie parallel algorihm i he early ieraios, bu he differece is egligible. Noe ha he compuaioal ime ad sigalig overhead of he ehaced sequeial algorihm does o scale well because hey are proporioal o K, he dimesio of x. By compariso, as he updae is parallelizable ad sigalig exchage is carried ou oly oce a each ime isa, he proposed algorihm achieves almos he same performace bu a a reduced cos of compuaioal ime ad sigalig overhead ha he ehaced sequeial algorihm cf. Secio V-A. We oe ha he esimaio wih sparsiy regularizaio performs beer ha he classic RLS approach idicaed by he magea curve, especially whe is small. This ca be explaied by he fac ha a prior iformaio of he sparsiy of he sigal x is exploied. The proposed algorihm performs beer ha he SPARLS algorihm wih opimal parameers 7] idicaed by he blue curve wih reversed riagular. However, o obai he opimal parameers, he maximum eigevalue of G mus be compued, which is a compuaioal prohibiive as i large-scale problems. If we use a subopimal parameer rg isead of he opimal parameer λ max G, he he performace of SPARLS wih subopimal parameers idicaed by he blue curve wih riagular deerioraes sigificaly, especially whe K is large, cf. Fig. b. We use he same choice of free parameers e.g., sepsize ad regularizaio gai for he rucaed gradie algorihm idicaed by he he gree i boh seigs K = 00 ad K = 500.

11 300 IEEE TRANSACTIONS ON SIGNAL AND INFORMATION PROCESSING OVER NETWORKS, VOL., NO. 3, SEPTEMBER 06 Fig.. Relaive square error for recursive esimaio of ime-varyig sigals. a Sigal dimesio: K = 00. b Sigal dimesio: K = 500. Fig. 3. Compariso of origial sigal ad esimaed sigal a differe ime isa: = 00 i he upper plo ad = 000 i he lower plo. I is observed from he compariso of Fig. a ad b ha he rucaed gradie algorihm 8] is sesiive o he choice of free parameers ad o geeral rule applies o all problem parameers. Furhermore, i coverges slowly because i is esseially a gradie mehod. By compariso, o preuig is required i he proposed algorihm ad simple closed-form expressios exis for each updae. The proposed algorihm is easy o use i pracice ad robus o chages i problem parameers. The precisio of he esimaed sigal by he proposed olie parallel algorihm afer 00 ad 000 ime isa, respecively is show eleme-wise i Fig. 3 whe K = 00. Give 00 measuremes, we observe from he upper plo of Fig. 3 ha he proposed olie parallel algorihm ca accuraely esimae he por of x, while as expeced, he esimaed sigal based o he RLS algorihm is o sparse. Whe he umber of measuremes is icreased o 000, we ca see from he lower Fig. 4. Weigh facor i ime- ad orm-weighed sparsiy regularizaio. plo of Fig. 3 ha he value of x is accuraely esimaed by he proposed olie parallel algorihm. The same observaio holds for he RLS algorihm as well because x rls x. C. Weigh Facor i Time- ad Norm-Weighed Sparsiy Regularizaio I Fig. 4 we simulae he weigh facor W μ x rls, versus he ime isa i ime- ad orm-weighed sparsiy regularizaio, where K = 00, =is used i he upper plo ad =i he lower plo. The parameers are he same as i he previous simulaio examples, excep ha μ =/ 0.4 ad x are geeraed such ha he firs 0. K elemes where 0. is he proporio of ozero elemes of x are ozero while all oher elemes are zero. The weigh facors of oher elemes are omied because hey exhibi similar behavior as he oes ploed i Fig. 4. As aalyzed, W μ w rls,, he weigh facor of he firs eleme, where x 0, quicly coverges o zero, while W μ w rls,, he weigh facor of he eleveh

12 YANG e al.: ONLINE PARALLEL ALGORITHM FOR RECURSIVE ESTIMATION OF SPARSE SIGNALS 30 VI. CONCLUDING REMARKS I his paper, we have cosidered he recursive esimaio of sparse sigals ad proposed a olie parallel algorihm wih provable covergece. The algorihm is based o approximae opimizaio bu i coverges o he exac soluio. A each ime isa, all elemes are updaed i parallel, ad boh he updae direcio ad he sepsize ca be calculaed i aalyical expressios. The proposed simplified miimizaio sepsize rule is well moivaed ad easily implemeable, achieves a good radeoff bewee complexiy ad covergece speed, ad avoids he commo drawbacs of he sadard sepsizes used i lieraure. Simulaio resuls have demosraed he oable improveme i covergece speed over sae-of-he-ar echiques. Our resuls show ha he loss i covergece speed compared wih he bechmar where he LASSO problem is solved exacly a each ime isa is egligible. We have also cosidered umerically he recursive esimaio of ime-varyig sigals where he heoreical covergece does o ecessarily hold, ad he proposed algorihm performs beer ha sae-of-he-ar algorihms. Fig. 5. Relaive square error for recursive esimaio of ime-varyig sigals. APPENDIX A PROOF OF THEOREM 5 Proof: I is easy o see ha L ca be divided io he differeiable par f x ad he odiffereiable par h x: L x =f x+h x, eleme, where x =0, quicly coverges o oe, maig he overall weigh facor moooically decreasig, cf. 9. Therefore he proposed algorihm ca readily be applied o he recursive esimaio of sparse sigals wih ime- ad orm-weighed regularizaio. D. Esimaio of he Time-Varyig Sigal Whe he sigal o be esimaed is varyig, he heoreical aalysis of he proposed algorihm is o valid aymore, bu we ca es umerically how he proposed algorihm performs compared wih he olie sequeial algorihm. The ime-varyig uow sigal is deoed as x R 00, ad i is chagig accordig o he followig law: x +, = αx, + w,, where w, CN0, α for ay such ha x, 0, wih α =0.99 ad β =0.9. I Fig. 5, he relaive square error x x / x is ploed versus he ime isa. Despie he lac of heoreical aalysis, we observe he esimaio error of he proposed olie parallel algorihm idicaed by he blac curve is almos as same as ha of he bechmar i which he LASSO problem is solved exacly idicaed by he red curve, so he approximae opimizaio is o a impedig facor for he esimaio accuracy. This is aoher advaage of he proposed algorihm over ] where a disribued ieraive algorihm is employed o solve 30 exacly, which ieviably icurs a large delay ad exesive sigalig. f x xt G x b T x, h x μ x. 3a 3b We also use f x; x o deoe he smooh par of he objecive fucio i 9: f x; x G x r x + c x x. 33 Fucios f x; x ad f x are relaed accordig o he followig equaio: f x ; x =f x, x + c x x, 34 from which i is easy o ifer ha f x ; x = f x. The from he firs-order opimaliy codiio, h x has a subgradie ξ h ˆx a x =ˆx such ha for ay x : x ˆx f ˆx Now cosider he followig equaio: ; x +ξ 0,. 35 L x + L x = L x + L x +L x L x. 36 The res of he proof cosiss of hree pars. Firsly we prove i Par I ha here exiss a cosa η>0such ha L x + L x η ˆx x. The we show i Par ha he sequece { L x + } coverges. Fially we prove i Par 3 ha ay limi poi of he sequece { x } is a soluio of. Par Sice c mi c>0for all c mi is defied i Proposiio from Assumpio A4, i is easy o see from ha

13 30 IEEE TRANSACTIONS ON SIGNAL AND INFORMATION PROCESSING OVER NETWORKS, VOL., NO. 3, SEPTEMBER 06 he followig is rue: L x + γˆx x L x γ c λ maxg γ ˆx x, 0 γ. Sice λ max G is a coiuous fucio 47] ad G coverges o a posiive defiie marix by Assumpio A, here exiss a λ < + such ha λ λ max G for all. We hus coclude from he precedig iequaliy ha for all 0 λ : L x + γˆx x L x γ c λγ ˆx x. 37 I follows from 5, 6 ad 37 ha L x + f x + γ ˆx x + γ h x +γ h ˆx 38 f x + γˆx x + γh x +γh ˆx 39 L x γc λγ ˆx x. 40 Sice he iequaliies i 40 are rue for ay 0 γ, we se γ = mic/ λ,. The i is possible o show ha here is a cosa η>0 such ha L x + L x L x + L x η ˆx x. 4 Besides his, because of Sep 3 i Algorihm, x + is i he followig lower level se of L x: L 0 {x : L x 0}. 4 Because x 0 for ay x, 4 is a subse of { x : } xt G x b T x 0, which is a subse of L 0 {x : } λ maxg x b T x Sice G ad b coverges ad lim G 0, here exiss a bouded se, deoed as L 0, such ha L 0 L 0 L 0 for all ; hus he sequece {x } is bouded ad we deoe is upper boud as x. Par Combiig 36 ad 4, we have he followig: L + x + L x + L + x + L x + = f + x + f x + + h + x + h x + f + x + f x +, 44 where he las iequaliy comes from he decreasig propery of μ by Assumpio A3. Recallig he defiiio of f x i 3, i is easy o see ha where + f + x + f x + = l + x + l x y = l τ x +, τ = g T x. Taig he expecaio of he precedig equaio wih respec o {y +, g + } N =, codiioed o he aural hisory up o ime +, deoed as F + : F + = { we have x 0,...,x +, { g 0,...,g }, { y 0 E +f + x + f x + F +] = E l + x + F +] = E l + x + F +],...,y } }, E l τ x + F +] τ = l τ x +, 45 τ = where he secod equaliy comes from he observaio ha l τ x + is deermiisic as log as F + is give. This ogeher wih 44 idicaes ha E L + x + L x + F +] E f + x + f x + F +] E l + x + F +] l τ x + + τ = + E l + x + F +] l τ x +, τ =

14 YANG e al.: ONLINE PARALLEL ALGORITHM FOR RECURSIVE ESTIMATION OF SPARSE SIGNALS 303 ad E L + x + L x + F +]] 0 l + E + x + F +] l τ x + τ = + l E + x F +] l τ x, τ = 46 where x] 0 = maxx, 0, ad X i 46 wih X {x, x,...,} is he complee pah of x. Now we derive a upper boud o he expeced value of he righ had side of 46: ] E where E l + x F +] l τ x τ = = E y r T x + x T R 3 x ] E = E y b Ğ y + y ] +E τ = = τ = = τ = = b T x + ] b T x +E x T Ğ x ] E y y ] y τ, x T Ğ x ], E {y,g } y g ] y τ g τ, E g g g ] g τ T g. 47 The we boud each erm i 47 idividually. For he firs erm, sice y is idepede of x, E y ] = E y ] ] = E y E y ] σ 48 for some σ <, where he secod equaliy comes from Jese s iequaliy. Because of Assumpios A ad A, y has bouded momes ad he exisece of σ is he jusified by he ceral limi heorem 48]. For he secod erm of 47, we have E x b T x ] E b T x x ] b E ] T x. Similar o he lie of aalysis of 48, here exiss a σ < such ha E b T x ] b E ] T σ x x. 49 For he hird erm of 47, we have E x T Ğ x ] = E max λ Ğ ] x K ] = x max{λ E maxğ, λ mi Ğ } x E ] max{λ maxğ, λ mi Ğ } x K ] E λ Ğ = x E = Ğ ] r Ğ T σ 3 50 for some σ 3 <, where he firs equaliy comes from he observaio ha x should alig wih he eigevecor associaed wih he eigevalue wih larges absolue value. The combig 48 50, we ca claim ha here exiss σ σ + σ + σ 3 > 0 such ha E l E + x F +] ] l τ x σ. τ = I view of 46, we have E E L + x + L x + F +]] ] σ 0. 3/ 5 Summig 5 over, we obai = E E L + x + L x + F +]] ] <. 0 The i follows from he quasi-marigale covergece heorem cf. 4, Th. 6] ha { L x + } coverges almos surely. Par 3 Combiig 36 ad 4, we have L x + L x η ˆx x + L x L x. 5 Besides his, i follows from he covergece of { L x + } lim L x + L x =0, ad he srog law of large umbers ha lim L x L x =0.

15 304 IEEE TRANSACTIONS ON SIGNAL AND INFORMATION PROCESSING OVER NETWORKS, VOL., NO. 3, SEPTEMBER 06 Taig he limi iferior of boh sides of 5, we have { } 0 = lim if L x + L x lim if lim if + lim { η ˆx x } + L x L x { η ˆx x } { } L x L x = η lim ˆx x 0, so we ca ifer ha lim ˆx x =0. Sice 0 lim if ˆx x lim ˆx x =0, we ca ifer ha lim if ˆx x =0 ad hus lim ˆx x =0. Cosider ay limi poi of he sequece { x }, deoed as x. Sice ˆx is a coiuous fucio of x i view of 9 ad lim ˆx x =0,imusbelim ˆx = ˆx = x, ad he miimum priciple i 35 ca be simplified as x x f x +ξ 0, x, whose summaio over =,...,K leads o x x T f x +ξ 0, x. Therefore x miimizes L x ad x = x almos surely by Lemma. Sice x is uique i view of Assumpios A, he whole sequece {x } has a uique limi poi ad i hus coverges o x. The proof is hus compleed. ACKNOWLEDGMENT The auhors would lie o ha he reviewers whose commes have grealy improved he qualiy of he paper. REFERENCES ] Y. Yag, M. Zhag, M. Pesaveo, ad D. P. Palomar, A olie parallel algorihm for specrum sesig i cogiive radio ewors, i Proc. 48h Asilomar Cof. Sigals, Sys. Compu., 04, pp ] S. Barbarossa, S. Sardellii, ad P. Di Lorezo, Disribued deecio ad esimaio i wireless sesor ewors, i Academic Press Library i Sigal Processig, R. Chellappa ad S. Theodoridis, Eds. New Yor, NY, USA: Academic, 04, vol., pp ] S. Hayi, Cogiive radio: Brai-empowered wireless commuicaios, IEEE J. Sel. Areas Commu., vol. 3, o., pp. 0 0, Feb ] J. Miola ad G. Maguire, Cogiive radio: Maig sofware radios more persoal, IEEE Pers. Commu., vol. 6, o. 4, pp. 3 8, Aug ] R. Zhag, Y.-C. Liag, ad S. Cui, Dyamic resource allocaio i cogiive radio ewors, IEEE Sigal Process. Mag., vol. 7, o. 3, pp. 0 4, May 00. 6] Y. Yag, G. Scuari, P. Sog, ad D. P. Palomar, Robus MIMO cogiive radio sysems uder ierferece emperaure cosrais, IEEE J. Sel. Areas Commu., vol. 3, o., pp , Nov ] S. Hayi, D. Thomso, ad J. Reed, Specrum sesig for cogiive radio, Proc. IEEE, vol. 97, o. 5, pp , May ] S.-J. Kim, E. Dall Aese, ad G. B. Giaais, Cooperaive specrum sesig for cogiive radios usig Kriged Kalma filerig, IEEE J. Sel. Topics Sigal Process., vol. 5, o., pp. 4 36, Feb. 0. 9] F. Zeg, C. Li, ad Z. Tia, Disribued compressive specrum sesig i cooperaive mulihop cogiive ewors, IEEE J. Sel. Topics Sigal Process., vol. 5, o., pp , Feb. 0. 0] O. Mehaa ad N. D. Sidiropoulos, Frugal sesig: Widebad power specrum sesig from few bis, IEEE Tras. Sigal Process., vol. 6, o. 0, pp , May 03. ] S. M. Kay, Fudameals of Saisical Sigal Processig, Volume I: Esimaio Theory. Eglewood Cliffs, NJ, USA: Preice-Hall, 993. ] A. Sayed, Adapive Filers. Hoboe, NJ, USA: Wiley-Iersciece, ] G. Maeos, I. Schizas, ad G. Giaais, Disribued recursive leassquares for cosesus-based i-ewor adapive esimaio, IEEE Tras. Sigal Process., vol. 57, o., pp , Nov ] G. Maeos ad G. B. Giaais, Disribued recursive leas-squares: Sabiliy ad performace aalysis, IEEE Tras. Sigal Process., vol. 60, o. 7, pp , Jul. 0. 5] D. Agelosae, J. A. Bazerque, ad G. B. Giaais, Olie adapive esimaio of sparse sigals: Where RLS mees he l -orm, IEEE Tras. Sigal Process., vol. 58, o. 7, pp , Jul ] Y. Kopsiis, K. Slavais, ad S. Theodoridis, Olie sparse sysem ideificaio ad sigal recosrucio usig projecios oo weighed l balls, IEEE Tras. Sigal Process., vol. 59, o. 3, pp , Mar. 0. 7] B. Babadi, N. Kaloupsidis, ad V. Taroh, SPARLS: The sparse RLS algorihm, IEEE Tras. Sigal Process., vol. 58, o. 8, pp , Aug ] J. Lagford, L. Li, ad T. Zhag, Sparse olie learig via rucaed gradie, J. Mach. Lear. Res., vol. 0, pp , ] Y. Che ad A. O. Hero, Recursive l, group Lasso, IEEE Tras. Sigal Process., vol. 60, o. 8, pp , Aug. 0. 0] R. Tibshirai, Regressio shriage ad selecio via he : A rerospecive, J. Roy. Sais. Soc. B, Sais. Mehodol., vol. 58, o., pp , Ju ] M. A. T. Figueiredo, R. D. Nowa, ad S. J. Wrigh, Gradie projecio for sparse recosrucio: Applicaio o compressed sesig ad oher iverse problems, IEEE J. Sel. Topics Sigal Process., vol., o. 4, pp , Dec ] S.-J. Kim, K. Koh, M. Lusig, S. Boyd, ad D. Gorievsy, A ieriorpoi mehod for large-scale l -regularized leas squares, IEEE J. Sel. Topics Sigal Process., vol., o. 4, pp , Dec ] A. Bec ad M. Teboulle, A fas ieraive shriage-hresholdig algorihm for liear iverse problems, SIAM J. Imag. Sci., vol., o., pp. 83 0, Ja ] T. Goldsei ad S. Osher, The spli Bregma mehod for L-regularized problems, SIAM J. Imag. Sci., vol., o., pp , ] F. Facchiei, G. Scuari, ad S. Sagraella, Parallel selecive algorihms for ocovex big daa opimizaio, IEEE Tras. Sigal Process., vol. 63, o. 7, pp , Nov ] G. Maeos, J. A. Bazerque, ad G. B. Giaais, Disribued sparse liear regressio, IEEE Tras. Sigal Process., vol. 58, o. 0, pp , Oc ] D. P. Berseas ad J. N. Tsisilis, Parallel ad Disribued Compuaio: Numerical Mehods. Eglewood Cliffs, NJ, USA: Preice-Hall, ] S. Chouvardas, K. Slavais, Y. Kopsiis, ad S. Theodoridis, A sparsiy promoig adapive algorihm for disribued learig, IEEE Tras. Sigal Process., vol. 60, o. 0, pp , Oc. 0. 9] P. Di Lorezo ad A. H. Sayed, Sparse disribued learig based o diffusio adapaio, IEEE Tras. Sigal Process., vol. 6, o. 6, pp , Mar ] P. Di Lorezo, Diffusio adapaio sraegies for disribued esimaio over Gaussia Marov radom fields, IEEE Tras. Sigal Process., vol. 6, o., pp , Nov ] Y. Che, Y. Gu, ad A. O. Hero, Sparse LMS for sysem ideificaio, i Proc. IEEE I. Cof. Acous., Speech, Sigal Process., o. 3, pp , ] Y. Che, Y. Gu, ad A. O. Hero, Regularized leas-measquare algorihms, Tech. Rep., Ju. 00. Olie]. Available: hp://arxiv.org/abs/ ] Y. Liu, C. Li, ad Z. Zhag, Diffusio sparse leas-mea squares over ewors, IEEE Tras. Sigal Process., vol. 60, o. 8, pp , Aug ] Y. Yag, G. Scuari, D. P. Palomar, ad M. Pesaveo, A parallel decomposiio mehod for ocovex sochasic muli-age opimizaio problems, IEEE Tras. Sigal Process.,vol. 64, o., pp , Ju ] G. Scuari, F. Facchiei, P. Sog, D. P. Palomar, ad J.-S. Pag, Decomposiio by parial liearizaio: Parallel opimizaio of muli-age sysems, IEEE Tras. Sigal Process., vol. 6, o. 3, pp , Feb ] Z. Qua, S. Cui, H. Poor, ad A. Sayed, Collaboraive widebad sesig for cogiive radios, IEEE Sigal Process. Mag.,vol.5,o.6,pp.60 73, Nov ] S. Ghadimi ad G. La, Opimal sochasic approximaio algorihms for srogly covex sochasic composie opimizaio I: A geeric

16 YANG e al.: ONLINE PARALLEL ALGORITHM FOR RECURSIVE ESTIMATION OF SPARSE SIGNALS 305 algorihmic framewor, SIAM J. Opimizaio, vol., o. 4, pp , Nov ] S. Boyd, N. Parih, E. Chu, B. Peleao, ad J. Ecsei, Disribued opimizaio ad saisical learig via he aleraig direcio mehod of mulipliers, Foud. Treds Mach. Lear., vol. 3, o., pp., 0. 39] D. P. Berseas, Noliear Programmig. Belmo, MA, USA: Ahea Scieific, ] S. Boyd ad L. Vadeberghe, Covex Opimizaio. Cambridge, U.K.: Cambridge Uiv. Press, ] J. F. Surm, Usig SeDuMi.0: A Malab oolbox for opimizaio over symmeric coes, Opimizaio Mehods Sofw., vol., o. 4, pp , Ja ] J. Mairal, F. Bach, J. Poce, ad G. Sapiro, Olie learig for marix facorizaio ad sparse codig, J. Mach. Learig Res., vol., pp. 9 60, ] M. Razaviyay, M. Sajabi, ad Z.-Q. Luo, A sochasic successive miimizaio mehod for osmooh ocovex opimizaio, Mah. Program., Ju. 03. Olie]. Available: hp://arxiv.org/abs/ ] S. Sudaram ad C. Hadjicosis, Disribued fucio calculaio ad cosesus usig liear ieraive sraegies, IEEE J. Sel. Areas Commu., vol. 6, o. 4, pp , May ] W. H. Greee, Ecoomeric Aalysis, 7h ed. Eglewood Cliffs, NJ, USA: Preice-Hall, 0. 46] MOSEK, The MOSEK Opimizaio Toolbox for MATLAB Maual, Versio Olie]. Available: hps:// 47] R. A. Hor ad C. R. Johso, Marix Aalysis. Cambridge, U.K.: Cambridge Uiv. Press, ] R. Durre, Probabiliy: Theory ad Examples, 4h ed. Cambridge, U.K.: Cambridge Uiv. Press, 00. Yag Yag S 09 M 3 received he B.S. degree from he School of Iformaio Sciece ad Egieerig, Souheas Uiversiy, Najig, Chia, i 009, ad he Ph.D. degree from he Deparme of Elecroic ad Compuer Egieerig, The Hog Kog Uiversiy of Sciece ad Techology, Hog Kog. From November 03 o November 05, he had bee a Posdocoral Research Associae a he Commuicaio Sysems Group, Darmsad Uiversiy of Techology, Darmsad, Germay. He joied Iel Deuschlad GmbH as a Research Scieis i December 05. His research ieress iclude disribued soluio mehods i covex opimizaio, oliear programmig, ad game heory, wih applicaios i commuicaio ewors, sigal processig, ad fiacial egieerig. Marius Pesaveo M 00 received he Dipl.- Ig. ad M.Eg. degrees from Ruhr-Uiversiä Bochum, Bochum, Germay, ad McMaser Uiversiy, Hamilo, ON, Caada, i 999 ad 000, respecively, ad he Dr.-Ig. degree i elecrical egieerig from Ruhr-Uiversiä Bochum i 005. Bewee 005 ad 007, he was a Research Egieer a FAG Idusrial Services GmbH, Aache, Germay. From 007 o 009, he was he Direcor of he Sigal Processig Secio a MIMOo GmbH, Duisburg, Germay. I 00, he became a Assisa Professor for Robus Sigal Processig ad a Full Professor for Commuicaio Sysems i 03, Deparme of Elecrical Egieerig ad Iformaio Techology, Darmsad Uiversiy of Techology, Darmsad, Germay. His research ieress iclude robus sigal processig ad adapive beamformig, high-resoluio sesor array processig, muliaea ad muliuser commuicaio sysems, disribued, sparse, ad mixed-ieger opimizaio echiques for sigal processig ad commuicaios, saisical sigal processig, specral aalysis, ad parameer esimaio. He has received he 003 ITG/VDE Bes Paper Award, he 005 Youg Auhor Bes Paper Award of he IEEE TRANSACTIONS ON SIGNAL PROCESSING, ad he 00 Bes Paper Award of he CROWNCOM coferece. He is a Member of he Ediorial board of he EURASIP Sigal Processig Joural, a Associae Edior for he IEEE TRANSACTIONS ON SIGNAL PROCESSING. He is currely servig he secod erm as a Member of he Sesor Array ad Mulichael Techical Commiee of he IEEE Sigal Processig Sociey. Megyi Zhag S 09 M 3 received he B.Sc. degree from he Deparme of Elecroic Iformaio, Huazhog Uiversiy of Sciece ad Techology, Wuha, Chia, i 009, ad he Ph.D. degree from he Deparme of Elecroic ad Compuer Egieerig, The Hog Kog Uiversiy of Sciece ad Egieerig, Hog Kog, i 03. She is currely a Posdocoral Research Associae i he Deparme of Compuer Sciece ad Egieerig, The Chiese Uiversiy of Hog Kog, Hog Kog. Her research ieress iclude saisical sigal processig, opimizaio, ad machie learig wih applicaios i wireless commuicaios ad fiacial sysems. Daiel P. Palomar S 99 M 03 SM 08 F received he Elecrical Egieerig ad Ph.D. degrees from he Techical Uiversiy of Caaloia UPC, Barceloa, Spai, i 998 ad 003, respecively. He is a Professor i he Deparme of Elecroic ad Compuer Egieerig, Hog Kog Uiversiy of Sciece ad Techology HKUST, Hog Kog which he joied i 006. Sice 03, he is a Fellow of he Isiue for Advace Sudy a HKUST. He had previously held several research appoimes, amely, a Kig s College Lodo, Lodo, U.K.; Saford Uiversiy, Saford, CA, USA; Telecommuicaios Techological Ceer of Caaloia, Barceloa, Spai; Royal Isiue of Techology, Socholm, Swede; Uiversiy of Rome La Sapieza, Rome, Ialy; ad Priceo Uiversiy, Priceo, NJ, USA. His curre research ieress iclude applicaios of covex opimizaio heory, game heory, ad variaioal iequaliy heory o fiacial sysems, big daa sysems, ad commuicaio sysems. He has received he 004/06 Fulbrigh Research Fellowship, he 004 ad 05 coauhor Youg Auhor Bes Paper Awards by he IEEE Sigal Processig Sociey, he 00/03 bes Ph.D. prize i Iformaio Techologies ad Commuicaios by he UPC, he 00/03 Rosia Ribala firs prize for he Bes Docoral Thesis i iformaio echologies ad commuicaios by he Epso Foudaio, ad he 004 prize for he bes Docoral Thesis i advaced mobile commuicaios by he Vodafoe Foudaio. He is a Gues Edior of he IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING06 Special Issue o Fiacial Sigal Processig ad Machie Learig for Elecroic Tradig ad has bee a Associae Edior of IEEE TRANSACTIONS ON INFORMATION THEORY ad of IEEE TRANSACTIONS ON SIGNAL PROCESSING, a Gues Edior of he IEEE SIGNAL PROCESSING MAGA- ZINE 00 Special Issue o Covex Opimizaio for Sigal Processig, he IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS 008 Special Issue o Game Theory i Commuicaio Sysems, ad he IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS 007 Special Issue o Opimizaio of MIMO Trasceivers for Realisic Commuicaio Newors.