Boosted LMS-based Piecewise Linear Adaptive Filters

Transcription

1 016 4h European Sgnal Processng Conference EUSIPCO) Boosed LMS-based Pecewse Lnear Adapve Flers Darush Kar and Iman Marvan Deparmen of Elecrcal and Elecroncs Engneerng Blken Unversy, Ankara, Turkey {kar, Ibrahm Delbala Turk Telekom Communcaons Servces Inc., Isanbul, Turkey Suleyman Serdar Koza Deparmen of Elecrcal and Elecroncs Engneerng Blken Unversy, Ankara, Turkey Absrac We nroduce he boosng noon exensvely used n dfferen machne learnng applcaons o adapve sgnal processng leraure and mplemen several dfferen adapve flerng algorhms. In hs framework, we have several adapve consuen flers ha run n parallel. For each newly receved npu vecor and observaon par, each fler adaps self based on he performance of he oher adapve flers n he mxure on hs curren daa par. These relave updaes provde he boosng effec such ha he flers n he mxure learn a dfferen arbue of he daa provdng dversy. The oupus of hese consuen flers are hen combned usng adapve mxure approaches. We provde he compuaonal complexy bounds for he boosed adapve flers. The nroduced mehods demonsrae mprovemen n he performances of convenonal adapve flerng algorhms due o he boosng effec. I. INTRODUCTION Boosng s consdered as one of he mos mporan ensemble learnng mehods n he machne learnng leraure [1] [3]. As an ensemble learnng mehod [4], boosng combnes several parallel runnng weakly performng algorhms o buld a fnal srongly performng algorhm, by fndng a lnear combnaon of weak learnng algorhms. However, sgnfcanly less aenon s gven o he dea of boosng n he adapve sgnal processng leraure. To hs end, our goal s a) o use he boosng noon n adapve flerng, b) derve several dfferen adapve flerng algorhms based on he boosng approach c) and demonsrae he nrnsc connecons of boosng wh he adapve mxure mehods [5] and daa reuse algorhms [6] wdely suded n he adapve sgnal processng leraure. Alhough boosng s nally nroduced n he bach seng [],.e., where algorhms boos hemselves over a fxed se of ranng daa, s laer exended o he onlne seng [7]. In he onlne seng, we neher need nor have a fxed se of ranng daa, however, he daa arrves one by one as a sream. Each newly arrvng daa s processed and hen dscarded whou any sorng. The onlne seng s naurally movaed by many real lfe applcaons especally for he ones nvolvng bg daa, where here s no enough sorage space avalable or he consrans of he problem requre nsan processng [8]. However, for our purposes, he onlne seng s especally mporan snce s drecly akn o adapve flerng framework where he sreamng or sequenally arrvng daa s used o adap he nernal parameers of he fler, eher o adapvely learn he underlyng model or o rack he nonsaonary daa sascs [9]. Specfcally, we have m parallel runnng adapve flers ha receve he npu vecors sequenally one by one. Each adapve algorhm can use a dfferen updae, such as he recursve leas squares RLS) updae or leas-mean squares LMS) updae. Afer recevng he npu vecor, each algorhm produces s oupu and hen calculaes s nsananeous error afer he observaon s revealed. These updaes are performed for all he m consuen flers n he mxure. However, n he onlne boosng approaches, hese adapaons a each me proceed n rounds from op o boom, sarng from he frs adapve fler o he las one o acheve he boosng effec [10]. Furhermore, unlke he usual mxure approaches [5], he updae of each adapve fler depends on he prevous adapve flers n he mxure. Based on he performance of he flers from 1 o k on he curren x,d ) par, he k1)h fler may gve more or less emphasze o x,d ) par n s adapaon n order o recfy he msake of he prevous adapve flers. Ths dea s clearly relaed o he adapve mxure algorhms wdely used n he sgnal processng leraure. However, unlke he mxure mehods, he updaes of he consuen flers are no ndependen n boosng mehods. We mplemen our boosng algorhms on pecewse lnear flers, snce such flers delver a sgnfcanly superor performance han lnear flers, wh a comparable complexy [11]. To hs end, we apply he boosng noon o several parallel runnng pecewse lnear LMS-based flers, and nroduce hree dfferen approaches o use he mporance weghs [10]. In he frs approach, weghed updaes, we use he mporance weghs drecly o produce ceran weghed LMS algorhms. In he second approach, daa reuse, we use he mporance weghs o consruc daa reuse adapve algorhms. The hrd approach, random updaes, uses he mporance weghs o decde wheher o updae he consuen flers, based on a random number generaed from a Bernoull dsrbuon wh he parameer equal o he wegh. The random updaes mehod can be effecvely used for bg daa processng [1], due o he reduced complexy. The oupu of he consuen flers s also combned usng a lnear fler o consruc he fnal oupu of he algorhm. The fnal combnaon fler s also updaed /16/$ IEEE 1593

2 016 4h European Sgnal Processng Conference EUSIPCO) usng he LMS algorhm [5]. II. PROBLEM DESCRIPTION AND BACKGROUND All vecors are column vecors and represened wh bold lower case leers. Marces are represened by bold upper case leers. For a vecor a or a marx A), a T or A T ) s he ranspose and TrA) s he race of he marx A. The me ndex s gven n he subscrp,.e., x s he sample a me. We work wh real daa for noaonal smplcy. We denoe he mean of a random varable x as E[x]. We sequenally receve r-dmensonal npu regressor) vecors {x } 1, x R r, and desred daa {d } 1, and esmae d by ˆd = f x ), 1) n whch, f.) s an adapve fler. A each me he esmaon error s gven by e = d ˆd, and s used o updae he parameers of he adapve fler. For presenaon purposes, we assume ha d [ 1, 1], however, our dervaons hold for any bounded bu arbrary desred daa sequences. For example, n he predcon problem d = x 1 and n he channel equalzaon applcaon {d } are he ransmed bs, where x s he receved daa from he channel. In our framework, we do no use any sascal assumpons on he npu vecors or on he desred daa such ha our resuls are guaraneed o hold n an ndvdual sequence manner [13]. Noe ha alhough nonlnear flers can ouperform lnear flers, hey usually undergo overfng, sably, and convergence ssues [11], [14]. Furhermore, nonlnear flers generally have hgher compuaonal complexes, whch lms her use n mos of he real-lfe applcaons [11], [14]. To overcome hese problems, pecewse lnear flers are proposed, whch mgae he overfng and sably ssues, whle offerng a comparable modelng performance o he nonlnear flers [11], [14]. Therefore, n hs paper, we are parcularly neresed n pecewse lnear flers, whch serve as an elegan alernave o lnear flers. We use a pecewse lnear adapve flerng mehod, such ha he desred sgnal s predced as ˆd = N =1 s,w T, x, where s, s he ndcaor funcon of he h regon,.e., s, = 1 f x R, and s, =0oherwse. Noe ha a each me, only one of he s, s s nonzero, whch ndcaes he regon n whch x les. Thus, f x R, we updae only he h lnear fler. As an example, consder -dmensonal npu vecors x, as depced n Fg. 1. Here, we consruc he pecewse lnear fler f such ha ˆd = f x )=s 1, w T 1,x s, w T,x = s w T 1,x 1 s )w T,x, ) Then, f s = 1 we shall updae w 1,, oherwse we shall updae w,, based on he amoun of he error, e. III. BOOSTED LMS ALGORITHMS As shown n Fg., a each eraon, we have m parallel runnng adapve flers wh esmang funcons f k), k) producng esmaes ˆd = f k) x ) of d, k = 1,...,m. Regon Regon 1 T T f x ) x w f x ) x w,, s 0 1, 1, s 1 Drecon vecor Separang hyper-plane Fg. 1: A sample -regon paron of he npu vecor.e., x ) space, whch s -dmensonal n hs example. s deermnes wheher x s n Regon 1 or no. As an example, f we use m lnear flers, ˆdk) = x T w k) s he esmae generaed by he kh consuen fler, and f we use pecewse lnear flers each of whch wh N dfferen regons), ˆdk) hese m flers are hen combned usng he lnear weghs = N =1 s,x T w,. The oupus of z o produce he fnal esmae as ˆd = z T y [5], where 1) m) y [ ˆd,..., ˆd ] T s he vecor of oupus. Afer he desred sgnal d s revealed, he m parallel runnng flers wll be updaed for he nex eraon. Moreover, he lnear combnaon coeffcens z are also updaed usng ordnary LMS mehod, as dealed laer n Secon III-D. Afer d s revealed, he consuen flers, f k), k = 1,...,m, are consecuvely updaed as shown n Fg. from op o boom,.e., frs k = 1 s updaed, hen, k = and fnally k = m s updaed. However, o enhance he performance, we use a boosed updang approach [], such ha, he k1)h fler receves a oal loss parameer, l k1) from he fler f k). l k1) = l k) o compue a wegh λ k), [ ) ] σ d f k) x ), 3). The oal loss parameer l k), ndcaes he sum of he dfferences beween he desred Mean Squared Error MSE), σ, and he squared error of he frs k 1 flers a me. Then, he dfference σ e k) ) s added o l k), o generae l k1), and l k1) s passed o [ he nex consuen fler as shown n Fg.. Here, ) ] σ d f k) x ) measures how much he kh consuen fler s off wh respec o he fnal MSE performance goal. For example, f d = fx )ν for some deermnsc nonlnear funcon f ) and ν s he observaon nose, hen σ can be seleced as an upper bound on he varance of he nose process ν. In hs sense, l k) measures how he consuen flers j =1,...,k are cumulavely performng on d, x ) par wh respec o he fnal performance goal. We hen use he wegh λ k) o updae he kh consuen fler wh one of he mehods weghed updaes, daa reuse, or random updaes, whch wll be explaned laer n he subsecons of hs secon. Our am s o make λ k) large f he frs k 1 consuen flers made large errors on d, so ha he kh fler gves more mporance o d, x ) 1594

3 016 4h European Sgnal Processng Conference EUSIPCO) x Inpu Vecor m) m) m) z ˆm) d - m m) f Adapve Fler Parameers Updae m) 1 m) e m) l Desred Sgnal d - e ) ) dˆ Combnng he resuls of all consuen flers Fnal Esmae ) z ˆ) d ) f Adapve Fler Parameers Updae 3) l - ) 1 ) l Combnaon Weghs ) e 1) 1) 1) z 1 ˆ1) d 1) f Adapve Fler Parameers Updae ) l - 1) 1 Fg. : The block dagram of a boosed adapve flerng sysem ha uses he npu vecor x o produce he fnal esmae ˆd. There are m consuen flers f 1),...,f m), each of whch s an adapve pecewse lnear fler ha k) generaes s own esmae ˆd. The fnal esmae ˆd s a lnear combnaon of he esmaes generaed by all hese consuen flers, wh he combnaon weghs z k) k) s correspondng o ˆd s. The combnaon weghs are sored n a vecor whch s updaed afer each eraon. A me he kh fler s updaed based on he values of λ k) and e k), and provdes he k1)h fler wh l k1) ha s used o compue λ k1). The parameer δ k) ndcaes he average Mean Squared Error MSE) of he kh fler over he frs esmaons, and s used n compung λ k). n order o recfy he performance of he overall sysem. We now explan how o consruc hese weghs, such ha 0 < λ k) 1. To hs end, we se λ 1) = 1, for all, and nroduce a weghng smlar { o [10], [15]. } We defne ) k) cl he weghs as λ k) mn 1, δ k) 1, where δ k) 1 ndcaes an esmae of he kh fler s MSE, and c 0 s a desgn parameer, whch deermnes he dependence of each fler updae on he performance of he prevous flers,.e., c =0corresponds o ndependen updaes, lke he ordnary combnaon of he flers [5], whle a greaer c ndcaes he greaer effec of he prevous flers performance on he wegh λ k) of he curren fler. Here, δ k) 1 s an esmae of he Weghed Mean Squared Error WMSE) of he kh consuen fler over {x } 1 and {d } 1. In he basc ) mplemenaon of onlne boosng [10], [15], 1 δ k) 1 s se o he classfcaon advanage of he weak learners [15], where hs advanage s assumed o be he same for all weak learners from k =1,...,m. In hs paper, o avod usng any a pror knowledge and o be compleely adapve, we choose as he weghed and hresholded MSE of he kh fler up δ k) 1 1) l 1) e o me 1 as δ k) = τ=1 [ ] ) λ k) τ 4 d τ f τ k) x τ ) τ=1 λk) τ, 4) [ ] where f τ k) k) x τ ) hresholds f τ x τ ) no he range [ 1, 1]. Ths hresholdng s necessary o assure ha 0 <δ k) 1, whch guaranees 0 <λ k) 1 for all k =1,...,m and. We pon ou ha δ k) can be calculaed recursvely. Regardng he defnon of δ k) and λ k),fhekh fler s good,.e., f δ k) s small enough, we wll pass less wegh o he nex flers, such ha hose flers can concenrae more on he oher samples. Hence, he flers can ncrease he dversy by concenrang on dfferen pars of he daa [5]. s are larger,.e., close o 1, f mos of he consuen flers, j =1,...,k, have errors larger han σ on d, x ), and smaller,.e., close o 0, f he par d, x ) s easly modeled by he prevous consuen flers such ha he flers k 1,...,m do no need o concenrae more on hs par. Based on hese weghs, we nex nroduce hree approaches o updae he consuen flers, whch are pecewse lnear flers explaned n Secon II updaed usng LMS algorhm. Furhermore, he weghs λ k) A. Drecly Usng λ s o Scale he Learnng Raes Snce 0 <λ k) 1, hese weghs can be drecly used o scale he learnng raes for he LMS updaes. When he kh fler receves he wegh λ k), updaes s fler coeffcens w k),,=1,...,n,as w k),1 = where 0 <μ k) I μ k) λ k) ) λ k) x x T w k) μ k), μk) λ k) x d, 5). Noe ha we can choose μ k) = μ for all k, snce he adapve algorhms work consecuvely from op o boom, and he h lnear fler of each dfferen consuen fler wll have a dfferen learnng rae μ λ k). B. A Daa Reuse Approach Based on he Weghs In hs scenaro, for updang w k),, we use he LMS updae = celkλ k) ) mes, where K s a fxed neger number, n k) o oban he w k) 1 as q 0) = w k),, ) q a) = I μ k) x x T w k) 1 = q n k) q a 1) μ k) x d,a=1,...,n k), ). 6) C. Random Updaes Based on he Weghs In hs scenaro, we use he wegh λ k) o generae random number from a Bernoull dsrbuon, whch equals 1 wh probably λ k), or equals zero wh probably 1 λ k). Then, f hs number s 1, we do he ordnary LMS updae on w k) oherwse we do no.,, 1595

4 016 4h European Sgnal Processng Conference EUSIPCO) Algorhm 1 Boosed LMS wh he proposed mehods 1: Inpu: x,d ) daa sream), m number of LMS pecewse lnear consuen flers runnng n parallel) and σ he desred MSE, upper bound on he error varance). : Inalze he regresson coeffcens w k),1 for each LMS fler and he combnaon coeffcens as z 1 = 1 m [1, 1,...,1]T and for all k se δ k) 0 =0. 3: for =1o T do 4: Receve he regressor daa nsance x 5: Compue he ndcaor funcons s k), for all k s 6: Compue he consuen fler oupus ˆdk) = N =1 sk), xt w k), 7: Produce he fnal esmae ˆd = z T 1) m) [ ˆd,..., ˆd ] T 8: Receve he rue oupu d desred daa) 9: λ 1) =1 l 1) =0 10: for k =1o m do 11: Updae he regresson coeffcens w k), by usng 1: e k) 13: λ k) 14: δ k) 15: Λ k) 16: l k1) LMS and he wegh λ k) nroduced algorhms n Secon III = d { =mn k) ˆd 1, δ k) 1 } ) k) cl [ ] ) = Λ k) 1 δk) 1 λk) 4 d f k) x ) =Λ k) 1 λk) Λ k) 1 λk) [ = l k) σ e k) 17: end for 18: z 1 = ) I μy y T z μy d 19: end for D. The Fnal Algorhm based on one of he ) ] Afer he desred daa d s revealed, we updae he consuen flers as well as he combnaon weghs z. To updae he combnaon weghs, we agan employ an LMS algorhm yeldng z 1 = I μy y T ) z μy d, 7) where μ>0 and y =[ˆd 1) m),..., ˆd ] T. The complee fnal algorhm s gven n Algorhm 1. IV. COMPLEXITY ANALYSIS In hs secon we compare he complexy of he proposed algorhms and fnd an upper bound for he weghs λ k). Suppose ha he npu vecor has a lengh of r,.e., x R r. Each consuen fler performs Or) compuaons o generaes s esmae, and requres Or) compuaons due o updang he lnear flers usng he LMS mehod n her mos basc mplemenaons). We derve he compuaonal complexy of usng he LMS updaes n dfferen boosng scenaros. Snce here are a oal of m consuen flers, all of whch are updaed n weghed samples mehod, hs mehod has a compuaonal cos of order Omr) per each eraon. However, n random updaes, a eraon, he kh fler wll or wll no be updaed wh probables λ k) and 1 λ k) respecvely. Hence, f E [ λ k) ] s upper bounded by λk) < 1, he average compuaonal complexy of he random updaes mehod, m wll be O λ k) r). In he Theorem, we provde suffcen k=1 consrans o have such an upper bound. Furhermore, we can use such a bound for he daa reuse mode as well. In hs case, for each fler f k), we perform he LMS updae λ k) K mes, resulng a compuaonal complexy of order K λ k) m Or)). k=1 The followng heorem deermnes he upper bound λ k) for E [ λ k) ]. Theorem: If he adapve flers converge and acheve a suffcenly small MSE accordng o he proof followng hs Theorem), he followng upper bound s obaned for λ k), gven ha σ s chosen properly, [ E λ k) where γ E ] λ k) = γ σ 1 ζ ln γ) [ ] [ ) ] δ k) 1 and ζ E e k). ) 1 k, 8) I can be sraghforwardly shown ha, hs bound s less han 1 for approprae choces of σ, and reasonable values for he MSE accordng o he proof. Ths heorem saes ha f we adjus σ such ha s achevable,.e., he adapve flers can provde a slghly lower MSE han σ, he probably of updang he flers n he random updaes scenaro wll decrease. Ths s of course our desred resul, snce f he flers are performng suffcenly well, here s no need for addonal updaes. Moreover, f σ s oped such ha he flers canno acheve a MSE equal o σ, he flers have o be updaed a each eraon, whch ncreases he complexy. Oulne of he proof: For smplcy, n hs proof, we have assumed ha c =1, however, he resuls are readly exended s are ndependen and dencally dsrbued..d) zero-mean Gaussan random varables wh varance ζ. I can be shown ha we acheve he saed upper bound n he Theorem, under he followng necessary and suffcen condons: o he general values of c. Assume ha e k) δ k) 1 1ζ ln ) σ )) < δ k) 1 1σ ) 4k 1), 9) and 1 ξ 1 )σ ) <ζ 1 ξ )σ < ), 10) 1 σ ln δ k) 1 1 σ ln δ k) 1 where ξ 1 = α 1 σ )α 1 σ ) α 4k 1)δ k) 1 )σ, k 1)δ k) 1 )σ 1596

5 016 4h European Sgnal Processng Conference EUSIPCO) ξ = α 1 σ ) α 1 σ ) α 4k 1)δ k) 1 )σ, k 1)δ k) 1 )σ and ) α 1ζ ln δ k) 1. V. EXPERIMENTS In hs secon, we demonsrae he effcency of he nroduced mehods n a nonsaonary envronmen. These expermens show ha our algorhms can successfully mprove he performance of sngle pecewse lnear flers, and n some cases, even ouperform he convenonal mxure mehod. We have consdered he case where he desred daa s generaed by a nonsaonary pecewse lnear model wh 3 regons. x = [x 1 x ] T s drawn from a jonly Gaussan random process, and hen scaled such ha x [0 1]. However, n hs expermen, we have dvded he oal daa nerval [0 T ] no 4 dsjon nervals, each of lengh T/4, and used a dfferen 3-regon model n each regon. In hs expermen, each boosng algorhm uses 5 consuen flers, each of whch uses a pecewse lnear fler over a -regon paron. The Accumulaed Squared Error ASE) performance of dfferen mehods are compared n Fg. 3. In he Fg. 3, PLMS, MIX, and BPLMS, respecvely show a sngle pecewse lnear LMS fler, he ordnary mxure mehod, and he boosed flers mehods. In addon, he suffxes WU, RU, and DR ndcae he weghed updaes, random updaes, and daa reuse mehods, respecvely. The learnng raes for he LMS-based algorhms are se o 0.0, and he desred MSE parameer σ s se o Also, he drecon vecor for he separang hyperplane s se o θ =[θ 1 θ θ 3 ] T. θ s conssed of hree random varables, each wh mean 1, o consruc random consuen flers. The resuls show he superor performance of our algorhms over he sngle pecewse lnear flers, as well as he mxure mehod, n hs hghly nonsaonary envronmen. Moreover, as shown n Fg. 3 he daa reuse mehod shows a beer performance relave o he oher boosng mehods. However, accordng o Table I he random updaes mehod has a sgnfcanly lower me consumpon, whch makes more desrable for bg daa applcaons. VI. CONCLUSION We nroduce he boosng concep, exensvely suded n machne learnng leraure, o adapve flerng conex, and propose hree dfferen boosng approaches, weghed updaes, daa reuse, and random updaes whch are applcable o dfferen adapve flerng algorhms. We show ha by hese approaches we can mprove he MSE performance of he convenonal LMS flers n pecewse lnear models, and we provde an upper bound for he weghs generaed durng he algorhm, whch lead us o a horough analyss of he complexy of hese mehods. We show ha he complexy of random updaes mehod s remarkably lower han oher wo approaches, whle he MSE performance does no degrade. Accumulaed Squared Error 10-1 Performance Comparson, LMS 10 - PLMS MIX BPLMS-WU BPLMS-RU BPLMS-DR Daa Lengh ) 10 4 Fg. 3: ASE performance TABLE I: Tme comparson of dfferen mehods seconds) LMS-MIX BPLMS-RU BPLMS-WU BPLMS-DR Therefore, he boosng usng random updaes approach can be effcenly appled o real lfe large scale problems. REFERENCES [1] R. E. Schapre and Y. Freund, Boosng: Foundaons and Algorhms, MIT Press, 01. [] Y. Freund and R. E.Schapre, A decson-heorec generalzaon of on-lne learnng and an applcaon o boosng, Journal of Compuer and Sysem Scences, vol. 55, pp , [3] D. L. Shresha and D. P. Solomane, Expermens wh adaboos.r, an mproved boosng scheme for regresson, n Expermens wh AdaBoos.RT, an mproved boosng scheme for regresson, 006. [4] R. O. Duda, P. E. Har, and D. G. Sork, Paern Classfcaon, John Wlley and Sons, 001. [5] S. S. Koza, A. T. Erdogan, A. C. Snger, and A. H. Sayed, Seady sae MSE performance analyss of mxure approaches o adapve flerng, IEEE Transacons on Sgnal Processng, 010. [6] S. Shaffer and C. S. Wllams, Comparson of lms, alpha-lms, and daa reusng lms algorhms, n Conference Record of he Seveneenh Aslomar Conference on Crcus, Sysems and Compuers, [7] N. C. Oza and S. Russell, Onlne baggng and boosng, n Proceedngs of AISTATS, 001. [8] L. Boou and O. Bousque, The radeoffs of large scale learnng, n NIPS, 008. [9] A. H. Sayed, Fundamenals of Adapve Flerng, John Wley and Sons, 003. [10] Shang-Tse Chen, Hsuan-Ten Ln, and Ch-Jen Lu, An onlne boosng algorhm wh heorecal jusfcaons, n ICML, 01. [11] N. D. Vanl and S. S. Koza, A comprehensve approach o unversal pecewse nonlnear regresson based on rees, IEEE Transacons on Sgnal Processng, vol. 6, no. 0, pp , Oc 014. [1] P. Malk, Governng bg daa: Prncples and pracces, IBM J. Res. Dev., vol. 57, no. 3-4, pp. 1:1 1:1, May 013. [13] S. S Koza and A. C. Snger, Unversal swchng lnear leas squares predcon, IEEE Transacons on Sgnal Processng, vol. 56, pp , Jan [14] Suleyman Serdar Koza, Andrew C. Snger, and Georg Zeler, Unversal pecewse lnear predcon va conex rees., IEEE Transacons on Sgnal Processng, vol. 55, no. 7-, pp , 007. [15] R. A. Servedo, Smooh boosng and learnng wh malcous nose, Journal of Machne Learnng Research, vol. 4, pp , 003. [16] A. Papouls and S. U. Plla, Probably, Random Varables, and Sochasc Processes, McGraw-Hll Hgher Educaon, 4 edon,