SIMPLE ALGORITHMS FOR FAST ADAPTIVE FILTERING. Francoise Beaufays and Bernard Widrow. Stanford University

Transcription

1 SIMPLE ALGORITHMS FOR FAST ADAPTIVE FILTERING Facoise Beaufays ad Bead Widow Depatmet of Electical Egieeig Stafod Uivesity ABSTRACT The LMS algoithm iveted by Widow ad Ho i 959 is the simplest, most obust, ad oe of the most widely used algoithms fo adaptive lteig. Ufotuately, its covegece ate is highly depedet upothe coditioig of the autocoelatio matix of its iputs: the highe the iput eigevalue spead, the slowe the covegece of the adaptive weights. This poblem ca be ovecome by pepocessig the iputs to the LMS lte with a xed data-idepedet tasfomatio that, at least patially, decoelates the iputs. Typically, the pepocessig cosists of a DFT o a DCT tasfomatio followed by apowe omalizatio stage. The esultig algoithms ae called DFT-LMS ad DCT-LMS. This techique is to be cotasted with moe taditioal appoaches such as ecusive least squaes algoithms, whee a estimate of the ivese iput autocoelatio matix is used to impove the lte covegece speed. Afte placig DFT-LMS ad DCT-LMS ito cotext, we popose thee dieet appoaches to explai the algoithms both ituitively ad aalytically. We discuss the covegece speed impovemet bought by these algoithms ove covetioal LMS, ad we mae a shot aalysis of thei computatioal cost. INTRODUCTION It is well ow fom the theoy of LMS (Widow, 985) that the mea squae eo of a adaptive lte taied with the LMS algoithm deceases ove time as a sum of expoetials whose time costats ae ivesely popotioal to the eigevalues of the autocoelatio matix of the iputs to the lte. This meas that small eigevalues ceate slow covegece modes i the MSE fuctio. This eseach was sposoed by EPRI ude cotact RP800-3, by NSF ude gat NSF IRI 9-53, ad by ONR ude cotct N J-787. Lage eigevalues, o the othe had, put a limit o the maximum leaig ate that ca be chose without ecouteig stability poblems (Widow,985). It esults fom these two couteactig factos that the best covegece popeties ae obtaied whe all the eigevalues ae equal, that is whe the iput autocoelatio matix is popotioal to the idetity matix. I this case, the iputs ae pefectly ucoelated ad have equal powe i othe wods, they ae samples of a white oise pocess. As the eigevalue spead of the iput autocoelatio matix iceases, the covegece speed of LMS deteioates. DFT-LMS ad DCT-LMS oe a solutio to this poblem. By pepocessig the iput data with a well-chose but xed tasfomatio that does ot deped o the iputs, ad with a simple powe omalizatio stage, they cause the iput eigevalues of the LMS lte to cluste aoud oe, ad speed up the covegece of the adaptive weights. Recusive least squaes algoithms also decoelate the iputs by pepocessig them, but they use to that eect a estimate of the ivese autocoelatio matix, which thus depeds o the actual iputs. The pefomace of the algoithms based o dataidepedet tasfomatios clealy depeds o the othogoalizig capabilities of the tasfom used. No geeal poof exists that demostates the supeioity of oe tasfom ove the othes. DFT-LMS st itoduced by Naaya (983) is the simplest algoithm of this family, both because of the expoetial atue of the DFT ad because scietists have developed a stog ituitio fo the Fouie tasfom. It is ou expeiece though that i most pactical situatios DCT-LMS pefoms much bette tha DFT-LMS. I additio, it has the advatage ove DFT-LMS to be eal valued. I this pape, we st compae the geeal philosophies of DFT/DCT-LMS ad ecusive least squaes (RLS) algoithms. We the explai, though thee dieet appoaches, the mechaisms of DFT-LMS ad DCT-LMS. We peset ew esults o covegece speed, ad coclude with a shot aalysis of computatioal cost.

2 DFT/DCT-LMS VS. RLS Iput z - z - z - By iteatively calculatig the ivese autocoelatio matix of the iput data ad usig it to compute the cuet lte weights, RLS implemets a exact least squaes solutio (Fali, 990 Hayi, 99). The majo advatages of RLS ove LMS ae its elatively low sesitivity to iput eigevalue spead, its fast covegece, ad the fact that, at least fo statioay iputs, the quality of its steady-state solutio eeps o impovig ove time. O the othe had, RLS sues fom poo tacig capabilities i ostatioay eviomets (Beshad, 989), fom high computatioal cost, ad fom lac of obustess ude cetai iput coditios. The computatioal cost ad obustess issues have bee addessed by eseaches i developig othe exact least squaes algoithms, the most famous of them beig the ecusive lattice lte algoithms. Lattice ltes typically equie less computatios pe iteatio tha RLS, but eve thei most obust foms ca peset stability poblems (Noth, 993). I additio, they ae log ad complicated to implemet. LMS is itisically slow because it does ot decoelate its iputs pio to adaptive lteig, but pepocessig the iputs with a estimate of the ivese iput autocoelatio matix i the fashio of RLS leads to the poblems cited above. The solutio we popose i the ext sectio cosists of pepocessig the iputs to the LMS lte with a xed tasfomatio that does ot deped o the actual iput data. The decoelatio will oly be appoximative, but the computatioal cost will emai vey low, ad the obustess ad tacig ability of LMS will ot be aected. DFT-LMS AND DCT-LMS The DFT-LMS ad DCT-LMS algoithms ae composed of thee simple stages (see Fig. ). Fist, the tapdelayed iputs ae pepocessed by a discete Fouie o cosie tasfom. The tasfomed sigals ae the omalized by the squae oot of thei powe. The esultig equal powe sigals ae iputted to a adaptive liea combie whose weights ae adjusted usig the LMS algoithm. With these two algoithms, the othogoalizig step is data idepedet oly the powe omalizatio step is data depedet (i.e. the powe levelsusedtoomalize the sigals ae estimated fom the actual data). x x - x - x -+ x -+ x Adaptive Weights Eo Discete Fouie / cosie Tasfom u (0) w (0) w () w () w (-) w (-) e u () u () Powe Nomalizatio _ + Output y d u (-) Desied Respose u (-) Figue : DFT-LMS ad DCT-LMS bloc diagam. A lteig appoach The -poit discete Fouie/cosie tasfom ca be see as a liea tasfomatio fom iputs x = (x x ; ::: x ;+) t to outputs u = (u (0)) u () ::: u ( ; )) t, whee u (i) is the i th output of the DFT/DCT at time (see Fig. ). Each output u (i) ca be expessed as a weighted sum of the iputs x ;l, fo l =0:: ;, that is as the covolutio of x with some discete impulse espose h i. I the case of the DFT, h i (l) = il ej 8 l =0:: ; The associated tasfe fuctio, ; e ;j! H i (!) = ; e;j! e j i epesets a badpass lte of cetal fequecy i=. TheDFTcathus be see as a ba of badpass ltes whose cetal fequecies spa the iteval [0 ]. Figue shows the of the tasfe fuctios H i (!) of a 8 8DFT.

3 Magitude(H (w)) 0 Magitude(H (w)) fequecy Magitude(H (w)) fequecy Magitude(H (w)) 3 fequecy Magitude(H (w)) 4 fequecy Magitude(H (w)) 5 fequecy fequecy Magitude(H (w)) Magitude(H (w)) 6 7 fequecy fequecy Figue : 8 8 DFT: s of the tasfe fuctios of a ba of badpass ltes.

4 At each time, the iput sigal x is decomposed ito sigals lyig i dieet fequecy bis. If the badpass ltes wee pefect, the outputs of the DFT would be pefectly ucoelated, but due to the pesece of side lobes (see Fig.) thee is some leaage fom each fequecy bi to the othes, ad thus some coelatio betwee the output sigals. I the case of the DCT, the i th impulse espose h i (l) is give by h i (l) = K i(l +=) i cos( ) 8 l =0:: ; whee K i == p foi = 0 ad fo i =:: ;. The coespodig tasfe fuctios ae give by H i (!) = K i cos( i ) ( ; e;j! )( ; (;) i e ;j! ) ; cos( i )e ;j! + : e;j! the oigial MSE ellipse, Fig. 3(b) shows it afte tasfomatio by a DCT matix. The shape of the ellipse is uchaged ad so ae the eigevalues of the autocoelatio matix. The powe omalizatio stage (cf. Fig. ) ca be viewed geometically as a tasfomatio that, while pesevig the elliptical atue of the MSE, foces it to coss all the coodiate axes at the same distace fom the cete. This opeatio is ot uitay ad it does modify the eigevalue spead. It almost always impoves it. The bette the aligmet of the hypeellipsoid with the coodiate axes, the moe eciet thepowe omalizatio will be (a hypeellipsoid pefectly aliged beig tasfomed i a hypesphee). Figue 3 shows the esult of powe omalizatio fo ou example. The ew ellipse is moe oud-shaped ad has lowe eigevalue spead. This is vey typical. They still epeset a ba of badpass ltes but with dieet cetal fequecies, dieet mai lobe ad side lobes, ad dieet leaage popeties. w w A geometical appoach The DFT-LMS ad DCT-LMS algoithms ca also be explaied ad illustated geometically. The DFT ad DCT matices deed by l F( l) = ej C( l) = K (l +=) cos( ) fo l =0:: ;, ae uitay matices (i.e. thei ows ae othogoal to oe aothe ad have euclidia om oe). Uitay tasfomatios pefom oly otatios ad symmeties, they do ot modify the shape of the object they tasfom. The mea squae eo of LMS is a quadatic fuctio of the weights (Widow, 985). Witig the MSE as a fuctio of the weights ad xig it to some costat value, we get a implicit quadatic fuctio of the weights that epesets a hypeellipsoid i the - dimesioal weight space. A uitay tasfomatio of the iputs otates the hypeellipsoid ad bigs it ito appoximate aligmet with the coodiate axes. The slight impefectio i aligmet is pimaily due to leaage i the tasfom, DCT o DFT. The idea is illustated fo a simple -weight case i Fig. 3. Figue 3(a) shows (a) w Figue 3: MSE hypeellipsoid (a) befoe tasfomatio, (b) afte DCT, (C) afte powe omalizatio. (c) w A aalytical appoach I ode to d pecise ifomatio o how well a give tasfom decoelates cetai classes of iput sigals, oe must set the poblem i a moe mathematical famewo. Tasfomig a sigal X by a matix T (the DFT o the DCT matix), tasfoms its autocoelatio matix R = E(XX t )ito B = E(TXX t T t ) = TRT t. The powe of TX ca be foud o the mai diagoal of B. Powe omalizig TX tasfoms its elemets TX i ito TX i =p Powe of (TXi ), havig the autocoelatio matix S =(diagb) ;= B (diagb) ;= : (b) w w

5 If T decoelated X exactly, B would be diagoal ad S would be the idetity matix I ad would have all its eigevalues equal to oe but sice the DFT ad the DCT ae ot pefect decoelatos, this does ot wo out exactly. Some theoy has bee developed i the past about the decoelatig ability of the DFT ad the DCT (see fo example Geade, 984 Gay, 977 Rao, 990) but the esults peseted i the liteatue ae i geeal too wea to allow us to ife aythig about the of the idividual eigevalues of S, which is ou mai iteest. Fo example, it has bee pove that the autocoelatio matix B obtaied afte the DFT o the DCT \asymptotically coveges" to a diagoal matix: \asymptotically" meaig as, the size of B, teds to iity, ad \coveges" beig udestood i a wea om sese. Fom this esult, we ca deduce that S will asymptotically covege to idetity as teds to iity. Howeve, we ca ot coclude aythig about the possible covegece of the idividual eigevalues of S to oe, eveythig depeds o how ad how fast S coveges to I. To obtai stoge esults, futhe assumptios ae ecessay, fo example egadig the class of iput sigals to be cosideed. Eigevalues ad eigevalue spead fo Maov- iputs Fist ode Maov sigals ae a vey geeal, pactical, ad yet simple class of sigals. They esult fom white oise passig though a sigle pole lowpass lte. Such a lte has a impulse espose that deceases geometically with a ate give by the lte pole. A Maov- iput sigal X =(x x ; ::: x ;+) t of paamete [0 ] has a autocoelatio matix R equal to R = 0 ::: ; ::: ; ; ; ::: C A Fo lage (theoetically fo tedig to iity), the miimum ad maximum eigevalues of a autocoelatio matix R ae give by the miimum ad maximum of the powe spectum of the sigal that geeated this autocoelatio (Geade, 984 Gay, 977). This esult is a Two matices covege to oe aothe i a wea om sese whe thei wea oms covege to oe aothe. The wea om of a matix is deed as the squae oot of the aithmetic aveage of its eigevalues. diect cosequece of the fact that R is Toeplitz. It ca easily be checed that i ou case the powe spectum of x is give by P (!) = +X l=; l e ;j!l = ; cos(!) + : Its maximum ad miimum ae espectively =( ; ) ad =( + ). The eigevalue spead of R thus teds to Eigevalue spead befoe tasfomatio = ( + ; ) : This eigevalue spead ca be extemely lage fo highly coelated sigals ( close to ). The autocoelatio S of the sigals obtaied afte tasfomatio by the DFT o the DCT ad afte powe omalizatio is ot Toeplitz aymoe, ad the pevious theoy ca ot be applied. The aalysis is futhe complicated by the fact that oly asymptotically do the eigevalues stabilize to xed s idepedet of, ad that powe omalizatio is a oliea opeatio. Successive matix maipulatios ad passages to the limit allowed us to pove the followig asymptotic esults (see Beaufays, 993, 994) fo moe details): Eigevalue spead afte DFT = + ; Eigevalue spead afte DCT = + : Note that with the DCT, the asymptotic eigevalue spead is eve highe tha! As a umeical example, let the coelatio be equal to The eigevalue spead befoe tasfomatio is 5, afte the DFT 39, afte the DCT.95. I this case, usig the DCT-LMS istead of LMS would speed up the lte weightcovegece by a facto oughly equal to 750. These esults com, fo a simple but vey pactical class of sigals, the high quality of the DCT as a sigal decoelato. Computatioal cost of DFT-LMS ad DCT-LMS I additio to thei fast covegece ad obustess, DFT-LMS ad DCT-LMS have the advatage of a vey low computatioal cost. The iputs x x ; ::: x ;+ beig delayed samples of the same sigal, the DFT-DCT ca be computed i O() opeatios. Fo the DFT, u (i) = ; X l=0 e j il x;l

6 = x + X l= e j il x;l ; e j i x; = e j i u;(i)+x ; x ;: The u (i)'s ca thus be foud by ao() ecusio fom the u ;(i)'s. This type of DFT is sometimes called slidig DFT. A simila O() ecusio ca be deived with moe algeba fo the DCT: u DCT (i) =u DCT ; (i)cos( i ) ; u DST (i)si( i )+ u DST (i) =u DST ; (i)cos( i )+ u DCT u DCT (i)si( i )+ cos( i )(x ; (;) i x;) si( i )(x ; (;) i x;): (i) is the i th output of the DCT, u DST (i) isthei th output of a DST (discete sie tasfom) deed exactly lie the DCT but eplacig \cos" by \si" (itelacig two ecusios is ecessay ad comes basically fom the fact that cos(a + b) = cos(a)cos(b) ; si(a)si(b)). The powe levels of the u (i)'s ca also be computed by a simple O() ecusio: P (i) =P ;(i)+u (i) whee P (i) = (powe of u (i))/( ; ), P;(i) is iitialized to zeo, ad [0 ] is a fogettig facto. Fially, the last step, the LMS adaptatio of the vaiable weights, is O(). The oveall algoithm is thus O(). CONCLUSION Fo the most pat, the DCT-LMS algoithm is supeio to the DFT-LMS algoithm. Both ae obust algoithms, cotaiig thee obust steps: tasfomatio, powe omalizatio (lie automatic gai cotol i a adio o TV), ad LMS adaptive lteig. These algoithms ae easy to pogam ad to udestad. They use a miimum of computatio, oly slightly moe tha LMS aloe. They wo almost as well as RLS but do't have obustess poblems. The lattice foms of RLS ae moe obust tha RLS, but they ae much moe dicult to pogam ad to udestad. All i all, the DFT-LMS ad DCT-LMS algoithms should d iceased use i pactical eal-time applicatios. REFERENCES Beaufays, F. ad Widow, B Tasfom domai adaptive ltes: a aalytical appoach. Submitted to IEEE Tas. o Sigal Poc. Beaufays, F Ph.D. Thesis, i pepaatio. Ifomatio Systems Lab., Stafod Uivesity, Stafod, CA. Beshad, N. ad Macchi, O Compaiso of RLS ad LMS algoithms fo tacig a chiped sigal. Poc. ICASSP. Glasgow, Scotlad: Fali, G. F. et al Digital Cotol of Dyamic Systems. Secod editio. Addiso-Wesley, Readig, MA. Gay, R.M.977. Toeplitz ad Ciculat Matices: II. Tech. epot Ifomatio Systems Lab., Stafod Uivesity. Geade, U. ad Szego, G Toeplitz foms ad thei applicatios. Secod editio. Chelsea Publishig Compay, New Yo. Hayi, S. 99. Adaptive Filte Theoy. Secod editio. Petice Hall, Eglewood Clis, NJ. Noth, R. C. et al A oatig-poit aithmetic eo aalysis of diect ad idiect coeciet updatig techiques fo adaptive lattice ltes. IEEE Tas. o Sigal Poc. 4, o.5(may): Rao, K. R. ad Yip, P Discete cosie tasfom. Academic Pess, Ic, Sa Diego, CA. Widow, B. ad Steas, S. D Adaptive Sigal Pocessig. Petice-Hall, Eglewood Clis, NJ. Facoise Beaufays eceived the bachelo degee i mechaical ad electical egieeig i 988 fom Uivesite Libe de Buxelles, Bussels, Belgium, ad the MSEE degee i 989 fom Stafod Uivesity. She is cuetly a PhD studet ad eseach assistat at Stafod Uivesity, woig ude the supevisio of Pof. B. Widow. He iteests iclude liea adaptive lteig, sigal pocessig, ad oliea eual etwos.