0.5. s(n) d(n) v(n) 0.5

Transcription

1 A Geeral Class of Noliear Normalized LMS-type Adaptive Algorithms Sudhakar Kalluri ad Gozalo R. Arce Departmet of Electrical ad Computer Egieerig Uiversity of Delaware, Newark, DE 976 kalluriee.udel.edu Abstract The Normalized Least Mea Square (NLMS) algorithm is a importat variat of the classical LMS algorithm for adaptive liear FIR lterig. It provides a automatic choice for the LMS step-size parameter which aects the stability, covergece speed ad steady-state performace of the algorithm. I this paper, we geeralize the NLMS algorithm by derivig a class of Noliear Normalized LMS-type (NLMS-type) Algorithms that are applicable to a wide variety of oliear lters. These algorithms are developed by choosig a optimal time-varyig step-size i the class of LMS-type adaptive oliear lterig algorithms. A auxiliary xed step-size ca be itroduced i the NLMS-type algorithm. However, ulike i the LMS-type algorithm, the bouds o this ew step-size for algorithm stability are idepedet of the iput sigal statistics. Computer simulatios demostrate that these NLMS-type algorithms have a potetially faster covergece tha their LMS-type couterparts. Itroductio The Least Mea Square (LMS) algorithm [] is widely used for adaptig the weights of a liear FIR lter that miimizes the mea square error (MSE) betwee the lter output ad a desired sigal. Cosider a iput (observatio) vector of N samples, x 4 = [x ; x ; : : : ; x N ] T, ad a weight vector of N weights, w 4 = [w ; w ; : : : ; w N ] T. Deotig the liear lter output by y = w T x, the lterig error, i estimatig a desired sigal d, is e 4 = y? d. The optimal lter weights miimize the MSE cost fuctio, J(w) 4 = Efe g, where Efg deotes statistical expectatio. I a eviromet of ukow or chagig sigal statistics, the LMS algorithm [] attempts to miimize the MSE by cotiually updatig the weights as w( + ) = w? e x; () This work was supported by the Natioal Sciece Foudatio uder Grat MIP where > is the so-called step-size of the update. The computatioal simplicity of the LMS algorithm has made it a attractive choice for several applicatios i liear sigal processig. However, it suers from a slow rate of covergece. Further, its implemetatio requires the choice of a appropriate step-size which aects the stability, steady-state MSE ad covergece speed of the algorithm. The stability regio for mea-square covergece of the LMS algorithm is give by < < ( = trace(r)) [, ], where R = 4 Efxx T g is the autocorrelatio matrix of the iput vector x. Whe the iput sigal statistics are ukow or time-varyig, it is dicult to choose a step-size that is guarateed to lie withi the stability regio. The so-called Normalized LMS (NLMS) algorithm [] addresses the problem of step-size desig i () by choosig a time-varyig step-size that miimizes the ext- 4 step MSE, J + = Efe ( + )g. After icorporatig a auxiliary xed step-size ~ >, the NLMS algorithm is writte as w( + ) = w? ~ e x; () kxk where kxk = 4 P N i= x i is the squared Euclidea orm of the iput vector x. The theoretical bouds o the stability of the NLMS algorithm are give by < ~ < []. Ulike the LMS step-size of (), the auxiliary step-size ~ is dimesioless ad the stability regio for ~ is idepedet of the sigal statistics. This allows for a easier step-size desig with guarateed stability of the algorithm. Further, the NLMS algorithm is kow to coverge much faster tha the LMS algorithm [3, 4]. We ca also iterpret () as a modied LMS algorithm, where the update term i () is divided (ormalized) by the squared-orm kxk, to esure stability uder large excursios of the iput vector x. I this paper, we geeralize the NLMS algorithm of () by derivig a class of Noliear Normalized LMS-type (NLMS-type) algorithms that are applicable to a wide variety of oliear lter structures. Although liear lters are useful i a umber of applicatios, several practical situatios require oliear processig of the sigals

2 ivolved i order to maitai a acceptable level of performace. Cosider a arbitrary oliear lter whose output is deoted by y y(w; x). The LMS algorithm of () ca be geeralized to yield the followig class of oliear LMS-type adaptive algorithms (see Sectio ): w i ( + ) = w i? e y ; i = ; ; : : : ; N: (3) Note that (3) ca be applied to ay oliear lter for which the derivatives y exist. The above algorithm iherits the mai problem of the LMS algorithm, amely, the diculty i choosig the step-size >. Ulike the liear case where step-size bouds are available, the complexity iheret i most oliear lters has precluded a theoretical aalysis of (3) to derive the stability rage for. There is thus a strog motivatio to develop automatic step-size choices to guaratee stability of the LMS-type algorithm. The NLMS-type algorithms developed i this paper address this problem. Just as the liear NLMS algorithm of () is developed from the classical LMS algorithm, we obtai a geeral oliear NLMS-type algorithm from the LMS-type algorithm of (3) by choosig a time-varyig step-size which miimizes the extstep MSE at each iteratio. As i the liear case, we itroduce a dimesioless auxiliary step-size whose stability rage has the advatage of beig idepedet of the sigal statistics. The stability regio could therefore be determied empirically for ay give oliear lter. We show through computer simulatios that these NLMS-type algorithms have, i geeral, a potetially faster covergece tha their LMS-type couterparts. Noliear LMS-type Adaptive Algorithms I this sectio, we briey review the derivatio of oliear LMS-type adaptive algorithms that have bee used i the literature for the optimizatio of several types of oliear lters. Cosider a geeral oliear lter with the lter output give by y y(w; x), where x ad w are the N-log iput ad weight vectors, respectively. The optimal lter weights miimize the mea square error (MSE) cost fuctio J(w) = Efe g = Ef(y(w; x)? d) g; (4) where d is the desired sigal ad e = y? d is the lterig error. The ecessary coditios for lter optimality are obtaied by settig the gradiet of the cost fuctio equal to zero: J(w) = E e y = ; i = ; ; : : : ; N: (5) Due to the oliear ature of y(w; x), ad cosequetly of the equatios i (5), it is extremely dicult to solve for the optimal weights i closed-form. The method of steepest descet is a popular techique which attempts to miimize the MSE by cotiually updatig the lter weights usig the followig equatio: w i ( + ) = w i? J ; i = ; ; : : : ; N; (6) where w i is the ith weight at iteratio, > is the step-size of the update, ad the ith compoet of the gradiet at the th iteratio is give from (5) by J = E e y : (7) I a situatio where the sigal statistics are either ukow or rapidly chagig (as i a ostatioary eviromet), we use istataeous estimates for the gradiet. To this ed, removig the expectatio operator i (7) ad substitutig ito (6), we obtai the followig class of oliear LMS-type adaptive algorithms: w i ( + ) = w i? e y : (8) Note that for a liear lter (y = w T x), we have y = x i, ad (8) reduces as expected to the LMS algorithm of (). As metioed i Sectio, there is a strog motivatio for the developmet of automatic step-size choices that guaratee the stability of (8); this is achieved by the NLMStype algorithms derived i the followig sectio. 3 Noliear Normalized LMStype (NLMS-type) Algorithms We derive the class of oliear NLMS-type algorithms by choosig a time-varyig step-size > i the LMS-type algorithm of (8). To this ed, we start by rewritig the steepest descet algorithm of (6), usig (5) to obtai w i ( + ) = w i? E e y : (9) Now, the ext-step MSE at the th iteratio is deed by J + 4 = J(w( + )) = Efe ( + )g; () where the ext-step lterig error e( + ) is e( + ) = y( + )? d( + ) y(w( + ); x( + ))? d( + ): () Note that J + depeds o the updated weight vector w( + ), which i tur is a fuctio of >. We obtai the NLMS-type algorithm from (9) by determiig the optimal step-size, deoted by o, that miimizes J + J + (): o 4 = arg mi > J +(): ()

3 To determie o, we eed a expressio for the derivative fuctio (=)J + (). Referrig to () ad (), we ca use the chai rule to write J +() = J + () ( + ) ( + ) : (3) To evaluate the expressios i (3), we rst dee the followig fuctios for otatioal coveiece (see (7)): j 4 =? =? E J w j e y We ca the rewrite the update i (9) as : (4) w i ( + ) = w i + i : (5) Usig (5), we obtai oe of the terms to be evaluated i (3) as ( + ) = j : (6) The other term i (3) ca be writte, usig (4), as J + () ( + ) J ( + ) =? j ( + ): (7) Returig to the derivative fuctio i (3), we use (6) ad (7) to obtai J +() =? j ( + ) j : (8) Before simplifyig (8) further, we ote from (5) that = ) w( + ) = w: (9) Thus, = correspods to quatities at time, while > correspods to quatities at time ( + ). Cosequetly, we otice i (8) that j (+) depeds o, while j does't. To emphasize this fact, dee the fuctios F j () It follows that 4 = j ( + ) =? E F j () = j =? E e( + ) y ( + ) : () e y : () Usig () ad (), we have the followig expressio for the derivative of J + (): J +() =? F j () F j (): () Due to the oliearity of the quatities i the above equatio (see () ad ()), it is very dicult to simplify () further i closed-form. We therefore resort to approximatig the fuctios F j () usig a rst-order Taylor's series expasio (liearizatio) about the poit =, assumig a small step-size > : F j () F j () + F j() = F j () + " F j() Usig this approximatio i (), we obtai J +()? 4 N X F j () + # = 3 : (3) Fj() F j () 5 : (4) Notice that this also implies a liearizatio of the derivative fuctio (=)J + (). This i tur is equivalet to approximatig the ext-step MSE J + () as a quadratic fuctio of. Uder these assumptios, the optimal stepsize o of () is foud by settig the (approximate) derivative of J + () to zero: o : J +() = : (5) = o I order to verify that (5) leads to a miimum, rather tha a maximum, of J + (), ote from () that J +() =? = F j () < : (6) Thus, J + () is (predictably) decreasig at =. Therefore, the quadratic approximatio of J + () ca be expected to attai its global miimum at some stepsize >. Usig (4) i (5), we obtai the followig closed-from, albeit approximate, expressio for this optimal step-size: where, from (), o? F j () =? E F j () F j() F j () e y ; (7) (8) is idepedet of, ad depeds oly o the sigal statistics at time. We see from (7) that our remaiig task is to evaluate F j (). The required expressio is derived

4 i [5], ad is give by F j() =? k= F k () E y e w k + y w k y ; (9) we omit the derivatio here due to lack of space. We ca ow substitute (9) ad (8) ito (7) ad obtai a expressio for the optimal step-size o. Note that (9) ad (8) ivolve statistical expectatios Efg. These expectatios are dicult to obtai i a eviromet of ukow or time-varyig sigal statistics. We therefore resort to usig istataeous estimates of these expectatios, just as i the derivatio of the covetioal (liear) LMS algorithm of () or of the oliear LMS-type algorithm of (8). To this ed, removig the expectatio operator i (9) ad (8), usig the resultig expressios i (7), ad performig some straightforward simplicatios, we obtai the followig expressio for the optimal step-size: o + E ; (3) y where E 4 = e y " k= X 4 N By makig the assumptio that y w k 3 y 5 # y w k : (3) jej ; (3) we ally obtai the followig simplied expressio for the optimal step-size: o : (33) y After icorporatig a auxiliary step-size ~ >, just as i the covetioal (liear) NLMS algorithm of (), we ca the write the time-varyig step-size, to be used i the steepest-descet algorithm of (9), as = ~ o ~ : (34) y Fially, o usig istataeous estimates by removig the expectatio operator i the steepest-descet algorithm of (9), we obtai the followig Noliear Normalized LMS-type Adaptive Filterig Algorithm: w i ( + ) = w i? ~ y This algorithm has several advatages: e y ; i = ; ; : : : ; N: (35) It is applicable to a wide variety of oliear lters; i fact, to ay oliear lter for which the lter output y is a aalytic fuctio of each of the lter weights w i (so that derivatives of all orders exist). The auxiliary step-size ~ is dimesioless ad the stability regio for ~ is idepedet of the sigal statistics. As a result, the stability regio could be determied empirically for ay particular oliear lter of iterest. This algorithm has a potetially faster covergece tha its LMS-type couterpart of (8), as demostrated by our simulatio results i Sectio 4. It ca also be iterpreted as a modicatio of the LMS-type algorithm of (8) i which the update term is divided (ormalized) by the Euclidea squaredorm of the set of values y ; i = ; ; : : : ; N, i order to esure algorithm stability whe these values become large i magitude. It is importat to ote the followig two approximatios used i derivig the NLMS-type algorithm of (35): Liearizatio of the fuctio F j () deed i () about the poit = (see (3)); this approximatio is valid oly for small values of the step-size. Note that this is ot, at least directly, a restrictio o the auxiliary step-size ~. The assumptio that jej (see (3) ad (3)). Cosider ow the special case of the liear lter, for which we have y = w T x, leadig to y = x i ; i = ; ; : : : ; N. It is the easily see that (35) reduces predictably to the (liear) NLMS algorithm of (). A sigificat poit to ote here is that we do ot require ay of the approximatios (see (3) ad (3)) that were used to obtai (35); the derivatio i this case is exact. Ideed, whe the lter is liear, the fuctio F j () of () ca be show to be liear i, thus elimiatig the eed for the liearizatio approximatio. Further, the expressio E of (3) is idetically equal to zero for the liear lter, makig the approximatio (3) uecessary.

5 4 Simulatio Results I order to ivestigate the covergece of the oliear NLMS-type algorithm of (35), it was applied to the problem of adaptive highpass lterig of a sum of siusoids i a impulsive oise eviromet. For the oliear highpass lter, we chose the so-called Weighted Myriad Filter [6, 7], which has recetly bee proposed for robust sigal processig i impulsive oise eviromets. Give a N- log iput vector x ad a vector of real-valued weights w, the Weighted Myriad Filter (WMyF) output is give by y K (w; x) 4 = arg mi i= log[k + jw i j (? sg(w i )x i ) ]; where K is called the liearity parameter sice y K reduces to the (liear) P weighted mea P of the iput samples N as K! : y = w N j x j = w j: For ite K, however, the lter output depeds oly o the N-log lter parameter vector h = 4 w=k. The lter ca therefore be adapted by updatig the parameters h i ; i = ; ; : : : ; N usig the NLMS-type algorithm of (35), with the required expressio for y h i give by s d Figure : Clea sum of siusoids s (top), ad desired highpass compoet d (bottom)..5 where i = y h i v i? + jhi j v i ; = =? i ; jh j j?? jhj j v j? + jhj j v j ; v.5 ad v i = sg(h i ) y? x i ; i = ; ; : : : ; N: I our simulatios, the observed P sigal was give by x = s + v, where s = k= a k si(f k ) is the clea sum of siusoids. The additive oise process v was chose to have a zero-mea symmetric -stable distributio [8] with characteristic expoet = :6 ad dispersio = :. Impulsive oise is well-modeled by the heavy-tailed class of -stable distributios, which icludes the Gaussia distributio as the special case whe =. The characteristic expoet ( < ) measures the heaviess of the tails, ad the dispersio decides the spread of the distributio aroud the origi. Fig. shows a portio of the sigal s which cosists of siusoids at digital frequecies f = :8; f = :; ad f = :. The desired sigal, also show i the gure, is the highest frequecy compoet, f. The additive -stable oise sigal v is show i Fig.. As the gure shows, the chose oise process simulates low-level Gaussia-type oise as well as impulsive iterferece. The oliear LMS-type algorithm of (8) (Sectio ) ad the ormalized LMS-type algorithm of (35) (Sectio 3) were used to trai the weighted myriad lter to extract the desired highpass sigal d from the oisy observed sigal x. A step-size of = :5 was used Figure : Additive -stable oise sigal v (characteristic expoet = :6, dispersio = :). i the LMS-type algorithm; this pushed the algorithm close to the limits of its stability regio, while maitaiig a acceptable al MSE. The ormalized LMS-type algorithm was used with a auxiliary step-size ~ = :, which is its default value, correspodig to the optimal step-size at each iteratio step. Note that this implies a automatic step-size choice i the NLMS-type algorithm, without a eed for step-size desig. The al traied lters, usig both the adaptive algorithms, were successful i accurately extractig the high-frequecy siusoidal compoet. We do ot show the lter outputs (usig the traied lter weights) here sice they are very close to the desired sigal. The al MSE usig both the LMStype ad the NLMS-type algorithms was approximately the same (aroud :). This allows for a meaigful compariso of the covergece speeds of the two al-

6 J J Figure 3: MSE learig curves, LMS-type (top) ad NLMS-type (bottom). gorithms. Fig. 3 shows the learig curves (MSE as a fuctio of algorithm iteratios) for the LMS-type as well as the NLMS-type algorithms. The LMS-type algorithm coverges i about iteratios to a MSE below :. O the other had, the NLMS-type algorithm is about te times faster, covergig to the same MSE i just iteratios. The gure clearly idicates the dramatic improvemet i covergece speed whe employig the NLMS-type algorithm. Notice the values of the MSEs i these curves; the NLMS-type algorithm has a lower MSE at each iteratio step. This is expected sice the NLMS-type algorithm was derived to miimize the extstep MSE at each iteratio of the LMS-type algorithm. Refereces [] S. Hayki, Adaptive Filter Theory. Eglewood Clis, NJ: Pretice Hall, 99. [] V. Solo ad X. Kog, Adaptive Sigal Processig Algorithms: Stability ad Performace. Eglewood Clis, NJ: Pretice Hall, 995. [3] D. T. M. Slock, \O the covergece behavior of the LMS ad the ormalized LMS algorithms," IEEE Trasactios o Sigal Processig, vol. 4, pp. 8{ 85, Sept [4] M. Rupp, \The behavior of LMS ad NLMS algorithms i the presece of spherically ivariat processes," IEEE Trasactios o Sigal Processig, vol. 4, pp. 49{6, Mar [5] S. Kalluri ad G. R. Arce, \A geeral class of oliear ormalized LMS{type adaptive lterig algorithms," IEEE Trasactios o Sigal Processig. I preparatio. [6] S. Kalluri ad G. R. Arce, \Adaptive weighted myriad lter algorithms for robust sigal processig i -stable oise eviromets," IEEE Trasactios o Sigal Processig, vol. 46, pp. 3{334, Feb [7] S. Kalluri ad G. R. Arce, \Robust frequecy{ selective lterig usig geeralized weighted myriad lters admittig real{valued weights," IEEE Trasactios o Sigal Processig. I preparatio. [8] C. L. Nikias ad M. Shao, Sigal Processig with Alpha-Stable Distributios ad Applicatios. New York: Wiley, Coclusio I this paper, we geeralized the ormalized LMS (NLMS) algorithm (proposed for liear lterig) by derivig a class of oliear ormalized LMS-type (NLMStype) algorithms with guarateed stability, that are applicable to a wide variety of oliear lters. These algorithms were obtaied by choosig a optimal timevaryig step-size i the oliear LMS-type algorithm, such that the ext-step mea square error (MSE) is miimized at each iteratio. Thus, the problem of stepsize desig was elimiated. A auxiliary step-size ca be itroduced i the NLMS-type algorithm; however, the bouds o this step-size for algorithm stability are idepedet of the iput sigal statistics, ulike i the case of the LMS-type algorithm. Computer simulatios of oliear highpass lterig i impulsive oise demostrated that these NLMS-type algorithms coverge much faster tha their LMS-type couterparts.