Gradient Compared lp-lms Algorithms for Sparse System Identification

Transcription

1 Gradent Comared l-lms Algorthms for Sarse System Identfcaton Yong Feng 1,, Jasong Wu, Ru Zeng, Lmn Luo, Huazhong Shu 1. School of Bologcal Scence and Medcal Engneerng, Southeast Unversty, Nanjng 10096, Chna E-mal: Laboratory of Image Scence and echnology, the Key Laboratory of Comuter Netor and Informaton Integraton, Southeast Unversty, Nanjng 10096, Chna E-mal: Abstract: In ths aer, e roose to novel -norm enalty least mean square (l -LMS) algorthms as sulements of the conventonal l -LMS algorthm establshed for sarse adatve flterng recently. A gradent comarator s emloyed to selectvely aly the zero attractor of -norm constrant for only those tas that have the same olarty as that of the gradent of the squared nstantaneous error, hch leads to the ne roosed gradent comared -norm constrant LMS algorthm (l GC-LMS). We exlan that the l GC-LMS can acheve loer mean square error than the standard l -LMS algorthm theoretcally and exermentally. o further mrove the erformance of the flter, the l NGC-LMS algorthm s derved usng a ne gradent comarator hch taes the sgn-smoothed verson of the revous one. he erformance of the lngc-lms s sueror to that of the lgc-lms n theory and n smulatons. Moreover, these to comarators can be easly aled to other norm constrant LMS algorthms to derve some ne aroaches for sarse adatve flterng. he numercal smulaton results sho that the to roosed algorthms acheve better erformance than the standard LMS algorthm and l -LMS algorthm n terms of convergence rate and steady-state behavor n sarse system dentfcaton settngs. Key Words: Least Mean Square Algorthm, Norm Constrant, Sarse System Identfcaton, Ne Gradent Comarator 1 INRODUCION Sarse systems, hose mulse resonses contan many near-zero coeffcents and fe large ones, are very common n ractce. For examle, sarse reless mult-ath channels, sarse acoustc echo ath, dgtal V transmsson channels, and so forth. he least mean square (LMS) algorthm [1 s the most dely used technque n alcatons le system dentfcaton (SI) hch lays an mortant role n the alcaton of adatve flterng. Hoever, the tradtonal LMS algorthm does not assume any structural nformaton about the system to be dentfed and thus erforms oorly both n terms of steady-state excess mean square error (excess MSE) and convergence rate [. Recently there emerges a grong research nterest n sarse system dentfcaton, for examle, sarse LMS algorthms th dfferent norm constrants, hch are manly motvated by research of the least absolute shrnage and selecton oerator (LASSO) [3 and comressve sensng (CS) [4. he famly of norm constrant LMS algorthms has become one of the man sarse LMS algorthms n adatve flterng durng the last fe years [5. hs or s suorted by the Natonal Basc Research Program of Chna under Grant 011CB707904, by the NSFC under Grants , , , and by the SRFDP under Grants and , the Project-sonsored by SRF for ROCS, SEM, and by Natural Scence Foundaton of Jangsu Provnce under Grant BK0139 and by Qng Lan Project. Many norm constrant LMS algorthms have been roosed so far, for nstance, l1-norm enalty LMS (l1-lms) [1, 6, l0-norm enalty LMS (l0-lms) [7, 8 and l-norm enalty LMS (l-lms) [9, 10, here the corresondng l1, l0 and l norms are ncororated nto the cost functon of the standard LMS algorthm resectvely, to ncrease the convergence seed and decrease the MSE as ell. Hoever, these algorthms aly the norm constrant of zero attractors to all the eght tas of the unnon system n general, leadng to a slong don of the convergence for those tas ho have the same olarty as the gradent of the squared error [11. Fortunately, t can be mroved by usng a gradent comactor that e ll shon n the later sectons. o c off the lmtaton above, e roose a gradent comared -norm enalty LMS algorthm (lgc-lms) as ell as ts mroved verson the lngc-lms, as sulements of the conventonal l-lms algorthm. Numercal smulatons sho that the roosed algorthms acheve better erformance than the standard LMS and l-lms algorthms n sarse system dentfcaton settngs. In addton, aher and Vorobyov have demonstrated that the l-lms s sueror to other norm constrant LMS algorthms n convergence behavor under certan condtons [9. he aer s organzed as follos: In Secton II, e roose the lgc-lms and ts extenson the lngc-lms algorthm for sarse systems n detals, after a bref reve of the standard LMS and l-lms algorthms. hen the numercal smulatons are gven n Secton III to nvestgate

2 the erformance of the to roosed algorthms. Fnally, the aer s concluded n Secton IV. PROPOSED ALGORIHMS hroughout ths aer, matrces and vectors are denoted by boldface uer-case letters and boldface loer-case letters, resectvely, hle varables and constants are n talc loer-case letters. he suerscrts ( ) reresents the transose oerators, and E[ denotes the exectaton oerator..1 Reveng standard LMS and l-lms algorthms Let y be the outut of an unnon system th an addtonal nose n at tme, hch can be rtten as y x n, (1) here the eght of length N s the mulse resonse of the unnon system, x reresents the nut vector th covarance R, defned as x [ x, x,, x N, and n s a statonary nose th zero mean and varance. Gven the nut x and outut y n an unnon lnear system follong the above settngs, the LMS algorthm as roosed to estmate the eght vector decades ago. he cost functon J of the standard LMS algorthm s defned as here e y J e /, () x denotes the nstantaneous error and 1 N [,[,...,[ s the estmated eght of the system at tme. Note that the 1/ here s taen just for the convenence of comutaton. hus, usng gradent descent, the udate equaton s rtten as J e x 1 0 max, (3) here μ s the ste sze such that th max beng the maxmum egenvalue of R. For the sarse system dentfcaton n hch most of the tas n the eght vector are exactly or nearly zeros, the l-lms [9 algorthm has been roosed th the ne cost functon J, rtten as J e, (4), / here the norm s defned as N 1 / [ th 0 < < 1, and γ s a constant controllng the trade-off beteen the convergence seed and estmaton error. hus, the udate of the l-lms s then derved as 1 sgn( ) e 1 x, (5) here ρ = μγ s an mortant arameter hch eghts the -norm constrant and sgnfcantly affects the erformance of the algorthm, ε s a constant boundng the term and sgn(x) s the sgn functon, hch s zero for x = 0, 1 for x > 0 and -1 for x < 0. Comared to the standard LMS, the udate of the l-lms has an extra udate term hch attracts all the coeffcents n the eght to zeros and thus, accelerate the convergence. Hoever, t ll lead to a slo don for the convergence of those tas that are otmzed by the term ex n the udate equaton, snce t does not dstngush the dfferent sarsty levels of the system [11.. Proosed lgc-lms and lngc-lms algorthms In order to conquer the above lmtaton of the l-lms, nsred by [11, e ntroduce a gradent comarator for the l-lms to obtan a ne algorthm called the lgc-lms, hch selectvely emloys the zero attractor of norm on those tas that have the same olarty as the gradent of the squared nstantaneous error. he udate equaton for the lgc-lms s gven by 1 sgn( ) e 1 x here G e x dag sgn( ) sgn / G, (6) s a dagonal matrx. Furthermore, e also exlored an mroved verson for the lgc-lms, called lngc-lms, hch emloys a ne gradent comarator that selectvely zero-attract only tas that have the same olarty as the gradent of the mean squared error. hus ts udate s obtaned as 1 sgn( ) e 1 x D, (7) 1 here D sgn G S s the sgned mean verson of the G above, and t should be emhaszed that the nteger S (ractcally set to 5 or more) emloyed here s sgnfcant hose logc ll be exlaned belo. he roles of gradent comarators G and D are elaborated as follos: Let [ be the th element of the real eght of the unnon system, such that [ 0, [ 0 and [ 0 are all the three ossbltes for the range of value [. Frstly, e consder the case [ 0 : (1) If [ [, then E[ ex E[( x n x ) x 0, and thus, [ G, th a hgh robablty of at least 50% hch deends on the sgnal nose rato (SNR), hereas, [ D, holds u th much hgher robablty due to the emloyed trc of sgned mean method, hch taes the exectaton to ncrease the lelhood of the equaton [ D, n ths case or [ D, 0 belo as much as ossble. Practcally e set the value of S to be 5 or more to made t. () When [ [, f [ 0, e have [ G, th a hgh robablty and [ D, th a much hgher robablty. Smlarly, e also have [ G, 0 and [ D, 0 th hgh and much hgher robabltes resectvely, f [ 0. Secondly, the same logc reasonng ll also aly to the case [ 0. Fnally,

3 hen [ 0, e have [ 0 or [ 0, [ G, and [ D, th both hgh but dfferent aforementoned robabltes are alays concluded. he lgc-lms or lngc-lms algorthm ors n the same ay as the l-lms hen [ G, or [ D, holds u,.e., the norm constrant s emloyed to attract the tas to zeros n ths case; And they ll be deduced to the standard LMS algorthm hen [ G, 0 and [ D, 0, hch neutralze the functon of the zero attractor of the l-lms. Addtonally, t should be underlned that the global convergence and consstency of the lgc-lms and lngc-lms reman roblematc, hch s nherted from the l-lms hose cost functon s not guaranteed to be convex. he Pseudo-codes for the lngc-lms algorthm s gven n able 1. Furthermore, e can also derve some other gradent comared norm-constrant LMS algorthms ncludng the l1-lms, l0-lms and some of ther varants. Added to ths, t s also orth mentonng that the varable ste sze (VSS) s non to have loer steady-state error as ell as faster convergence. hus, as a further extenson, the VSS lgc-lms and VSS lngc-lms can be easly derved n the same ay as t has been added to the -norm-le LMS n [10. 3 SIMULAIONS Numercal smulatons are carred out for several scenaros n ths secton to nvestgate the erformances of the roosed lgc-lms as ell as the lngc-lms algorthm n terms of the steady-state mean square devaton (MSD, defned as MSD E[ ) and convergence seed, and ther results are comared th the l-lms and the standard LMS algorthm n the settngs of sarse system dentfcaton th dfferent sarsty levels. Addtonally, the sueror erformance of the l-lms to other sarsty-aare modfcatons of LMS algorthms that are beyond the norm constrants, has been shon n Ref. [9, hch, to be bref, s not detaled here, nor s the contrast th l1gc-lms n Ref. [11 as a modfcaton of l1-lms, due to the arameter deendence roblem n comarson th the l-lms algorthm [ Examle 1: sarse system th hte Gaussan nut In the frst exerment, e estmate a sarse unnon tme-varyng system of 16 tas th 1, 4 or 8 tas that are assumed to be nonzeros, mang the sarsty rato (SR) be 1/16, 4/16 or 8/16, resectvely. he ostons of nonzero tas are chosen randomly and the values are 1 s or -1 s randomly. Intally, e set SR = 1/16 for the frst 500 teratons, and after that e have SR = 4/16 for the next 500 teratons, and then SR = 8/16 for the last 500 teratons hch leaves a sem-sarse system. he nut sgnal and observed nose are both assumed to be hte Gaussan rocesses of length 516 th zero mean and varances 1 and 0.01, resectvely,.e., the sgnal nose rato (SNR) s set to be 0 db. Other arameters are carefully selected as lsted n able. Note that e use the same ste sze μ for all the four flters and the same ρ, ε and for the three norm constrant LMS algorthms. All the smulatons are averaged over 00 ndeendent runs to smooth out ther MSD curves. Fg. 1 shos the MSD curves of the roosed lgc-lms and lngc-lms algorthms th resect to the number of teratons th dfferent sarsty levels,.e., SR = 1/16, 4/16 and 8/16 for the three teratve stages, comared th the standard LMS and l-lms algorthms. As shon n Fg. 1, e can see that n the very sarse case (SR=1/16) for the frst 500 teratons, the lngc-lms acheves the best erformance n terms of convergence seed and stable error, hle the lgc-lms ors a lttle orse than the l-lms, robably due to the aforementoned robablty roblem of G, It stll erform better than the standard LMS. Wth the sarsty decreasng (SR=4/16, 8/16 for next stages th 500 teratons each), the erformances of all the three -norm constrant LMS algorthms deterorate as exected. Hoever, the lngc-lms stll has better erformance than the l-lms and the lgc-lms, and the lgc-lms stll erforms better than the l-lms n these to cases hen the system s sem-sarse. hus, t can be concluded that the roosed lgc-lms and lngc-lms algorthms outerform the l-lms for a system th dfferent sarsty levels and hte Gaussan nut. able 1. Pseudo-codes of the l NGC-LMS Algorthm Gven Intal for end μ, ρ, N, S, ε,, x, L 0=zeros(N,1),,Y=zeros(1,L) = 1,,..., (L-N+1) y x e n y x 1 e 1 x D G dag sgn( ex ) sgn / 1 D sgn G S sgn( ) able. Parameters of the algorthms n the examle 1. Algorthms μ ρ ε S LMS [1 l -LMS [ / 0.05 l GC-LMS [b l NGC-LMS [c 5 [a~[c are for SR=1/16, 4/16 and 8/16, resectvely. Fg. shos the MSD curves of these algorthms tested th resect to dfferent sarsty levels (th fxed ρ = ), from hch e can see that the MSDs of all the three -norm constrant LMS algorthms ncrease th the SR, hereas the MSD of the standard LMS s stable. As observed from Fg.1, the lngc-lms has the loest MSD hen the system s sarse, then comes by the lgc-lms. It s mortant to note that the lngc-lms has loer MSD than the l-lms hatever the sarsty level of the system s, under the common settngs n ths smulaton. 1 Generally, 0.5 s the default value for n most l-lms related roblems th fxed values, hereas e have exlored and acheved a ne l-lms th dfferent methods of varable and better erformances, hch ll be secfed n the follong aer, due to the sace lmtatons.

4 able 3. Parameters of the algorthms n the examle. Fg. 1. MSD curves of dfferent algorthms th SNR = 0 db and SR=1/16, 4/16 and 8/16, resectvely.(hte Gaussan nut) Fg.. MSD curves of tested algorthms th fxed ρ = , dfferent sarsty levels and SNR = 0 db. 3. Examle : sarse system th correlated nut he unnon system n the second examle s the same as the revous one, excet that e change the stchng tmes for dfferent sarsty levels to the 3000th teraton and the 6000th teraton, resectvely. he nut s no a correlated sgnal hch s generated by x+1 = 0.8x + u and then normalzed to varance 1, here u s a hte Gaussan nose th varance And the varance of the observed nose s set to 10 1 n ths examle. Other arameter choces for all the algorthms tested are lsted n able 3. As t s set n the frst examle, e choose the same μ, ρ, ε and for all the tested algorthms f they are requred. Hoever, to solve the roblem underlned n the revous examle that the norm constrant LMS algorthms may reform orse than the standard LMS due to the choces of the arameter, dfferent -norm constrant eght ρ's are selected for dfferent sarsty levels n ths case, hch ould utlze the sarse constrants better, and thus acheve loer steady-state MSD values than that of the standard LMS algorthm. In addton, all the MSD curves of the smulatons are smoothed va 00 Monte-Carlo runs as ell. Algorthms μ ρ ε S LMS [1 l-lms [ / l GC-LMS [b l NGC-LMS [c 5 [a~[c are for SR=1/16, 4/16 and 8/16, resectvely. Fg. 3 shos the MSD curves of the algorthms tested for dfferent sarsty levels,.e., SR = 1/6 for the frst 3000 teratons, SR = 4/16 for the 3001th to 6000th teraton and SR=8/16 for the next 3000 teratons left. It can be seen from Fg. 3 that smlar erformance trends are observed as n the frst examle,.e., most of the observatons from Fg. 1 also hold true for ths case here the nut sgnal s related. For systems of dfferent sarsty levels, the lngc-lms and lgc-lms alays acheve the faster convergence and loer steady-state MSD than the l-lms. Moreover, the lngc-lms alays erforms better than the lgc-lms and then folloed by the l-lms n any case. Hoever, as the sarsty rato ncreases, the behavors of all the norm constrant LMS algorthms tested n ths aer hch are exected to be better than the standard LMS algorthm, deend strctly on the arameters ρ and the SNR of the nut. Secfcally, as SR ncreases from 1/16 to 8/16, the smaller ρ, and / or the smaller SNR n a certan range, the better erformances hch the -norm constrant LMS algorthms acheve, and the smaller erformance gas beteen the lngc-lms and lgc-lms, the lgc-lms and l-lms. Anyay, the lngc-lms erforms the best among these three -norm constrant LMS algorthms, and better than or equvalently to the standard LMS f roer arameters are selected as n the settngs of ths examle. Fg. 3. MSD curves of dfferent algorthms th SR=1/16 (ρ = ), 4/16 (ρ = ) and 8/16 (ρ = 10-5 ), resectvely. (correlated nut) 3.3 Examle 3: system of ECG-le mulse resonse In ths examle, e estmate a sarse system th an ECG-le mulse resonse (IR) of 56 tas, n hch 8 of them are nonzeros. Fg. 4 shos the ta vector of the system to be dentfed, hch s samled from an ECG sgnal and then rocessed nto a smler verson th most of the small tas set to zeros, to mae t sarse for our smulatons. he nut sgnal and addtonal nose are hte Gaussan

5 rocesses th varance 1 and 0.1, resectvely. Other arameters are selected n able 4 and all the smulatons are erformed 00 tmes. Note that e choose these arameters that are dfferent from the revous to examles to yeld better erformance n term of convergence seed and steady-state error lotted n MSD curves, hch s shon n Fg. 5. able 4. Parameters of the algorthms n the examle 3. Algorthms μ ρ ε S LMS l -LMS / l GC-LMS l NGC-LMS From Fg. 5, one can see that the revous observatons stll hold u n ths much longer sarse system, the roosed lngc-lms and lgc-lms outerform the l-lms and standard LMS algorthm th faster convergence seed and loer steady-state MSD. Fg. 4. he mulse resonse of the system n the examle 3. Fg. 5. MSD curves of dfferent algorthms th SNR = 10 db and SR=8/56. (ECG-le system) 4 CONCLUSION We roose to gradent comared -norm enalty LMS algorthms,.e., the lgc-lms and lngc-lms algorthm, as extensons of the conventonal l-lms algorthm hch s establshed for sarse adatve flterng recently. o gradent comarators are emloyed to selectvely aly the zero attractor n l-lms for only those tas that have the same olarty as that of the gradent of the squared nstantaneous error or mean squared error. Numercal smulaton results sho that the roosed algorthms yeld better erformances than those of the standard LMS and l-lms algorthms n sarse system dentfcaton settngs n term of convergence seed and steady-state mean square devaton. Summarly, e can conclude from the above statements n ths aer that the roosed lgc-lms and lngc-lms are actually to aroaches to eght the -norm constrant for the l-lms algorthm to select certan tas n order to erform better, and there are also some other methods to acheve ths. herefore, our future or ll stll focus on the choces of the arameters of the l-lms algorthm for sarse system dentfcaton, ncludng exlorng the relatonsh among the eght of norm constrant, the sarsty rato of the mulse resonse and the sgnal-nose rato of the nut, consderng an adatve arameter and ρ for dfferent sarsty levels and / or dfferent SNR, and develong ne l-lms algorthms for better dentfyng an unnon system hch s not assumed to be sarse or non-sarse, etc. REFERENCES [1 B. Wdro and S. D. Stearns, Adatve sgnal rocessng, Engleood Clffs, NJ, Prentce-Hall, Inc., [ Y. L. Chen, Y.. Gu, and A. O. Hero, Sarse LMS for system dentfcaton, IEEE Internatonal Conference on Acoustcs, Seech and Sgnal Processng, 009. [3 R. bshran, Regresson shrnage and selecton va the lasso: a retrosectve. Journal of the Royal Statstcal Socety: Seres B (Statstcal Methodology), Vol.73, No.3, 73-8, 011. [4 D. L. Donoho, Comressed sensng, IEEE rans. on Informaton heory, Vol.5, No.4, , 006. [5 B. K. Das, L. A. Azcueta-Ruz, M. Charaborty, et al, A comaratve study of to oular famles of sarsty-aare adatve flters, 4 th Internatonal Worsho on Cogntve Informaton Processng, 014. [6 K. Sh. and P. Sh, Convergence analyss of sarse LMS algorthms th l 1-norm enalty based on hte nut sgnal, Sgnal Processng, Vol.90, No.1, , 010. [7 Y.. Gu, J. Jn, and S. L. Me, l 0 Norm Constrant LMS Algorthm for Sarse System Identfcaton, IEEE Sgnal Processng Letters, Vol.16, No.9, , 009. [8 G. L. Su, J. Jn, Y.. Gu, et al, Performance Analyss of l 0 Norm Constrant Least Mean Square Algorthm, IEEE rans. on Sgnal Processng, Vol.60, No.5, 3-35, 01 [9 O. aher and S. A. Vorobyov, Sarse channel estmaton th l -norm and reeghted l 1-norm enalzed least mean squares, IEEE Internatonal Conference on Acoustcs, Seech and Sgnal Processng, 011. [10 M. L. Alyu, M. A. Alassm, and M. S. Salman, A -norm varable ste-sze LMS algorthm for sarse system dentfcaton, Sgnal, Image and Vdeo Processng, 1-7, 014. [11 B. K. Das and M. Charaborty, Gradent Comarator Least Mean Square Algorthm for Identfcaton of Sarse Systems th Varable Sarsty, APSIPA ASC, 011.