Blind Signal Separation Methods for Integration of Neural Networks Results

Transcription

1 Bnd Sgna Searaton Methods for Integraton of eura etwors Resuts Ryszard Szuu Warsaw Schoo of Economcs A.eodegosc 62, Warsaw PL Posa eefona Cyfrowa Ltd A. Jerozomse 8, Warsaw, PL Potr Wojewn Warsaw Schoo of Economcs A.eodegosc 62, Warsaw PL Posa eefona Cyfrowa Ltd A. Jerozomse 8, Warsaw, PL omasz Zabows Posa eefona Cyfrowa Ltd A. Jerozomse 8, Warsaw, PL Abstract - In ths aer t s roosed to ay bnd sgna searaton methods to mrove a neura networ redcton. Resuts generated by any regresson mode usuay ncude both constructve and destructve comonents. In case of a few modes, some of the comonents can be common to a of them. Our am s to fnd the bass eements and dstngush the comonents wth the constructve nfuence on the modeng quaty from the destructve ones. After rejectng the destructve eements from the modes resuts t s observed the enhancement of the resuts n terms of some standard error crtera. he vadty and hgh erformance of the concet s resented on the rea robem of energy oad redcton. Keywords: ensembe methods, regresson, neura networs, bnd sgna searaton. Introducton he neura networs, as we as the other regresson modes, try to reresent the deendency between nut data and target. o obtan roer resut, the cost functon of the networ arameters s otmzed [4, 2]. ycay, n rea robems many dfferent tyes and structures of neura networs are tested. After the earnng stage s fnshed we can evauate redcton resuts and choose the best neura mode [,3,6,9]. In ths standard methodoogy there aear some robems that shoud be consdered. Dfferent cost functons can ead to varous networ arameters as the otma and sometmes the choce of the functon s not obvous. here s no assurance that earnng rocess reay fnds the goba otmum of the cost functon [2,4]. Fnay, the term the best neura networ s not recse, because there est many evauaton crtera and dfferent error measures can ndcate dfferent modes as the best. It s qute ossbe that there est few modes of smar quaty or accordng to dfferent crtera each of the modes s evauated as the best. herefore, t seems natura to ntegrate and to use the nformaton generated by many neura networs what eads us to wde area of ensembe methods. Usuay soutons roose the combnaton of a few modes by mng ther resuts or arameters [5,3,22]. In ths aer we roose an aternatve concet based on mutdmensona decomostons [2]. ow et s assume that we have severa neura networs of accetabe redcton quaty. If dfferent networs gve reatvey good resuts t means that a of them are ocated cose to the target and the resuts of every mode ossess the constructve comonents as we as the destructve comonents and noses. We can assume that there est the eements common to a the modes. he common constructve comonents are assocated wth the unnown true vaue, of course. On the other hand the destructve comonents can be resent due to many reasons. Some of them ncude: mssng data, not recse arameter estmaton and dstrbuton assumtons. Our am s to dentfy the destructve eements and to emnate them what shoud gve us the modeng mrovement. o obtan such nterestng comonents we coect artcuar mode resuts together, treat them as mutvarate varabe and transform them wth one of the bnd sgna searaton methods. he fu rocess has the foowng stes:. Coectng the modes resuts together n one mutvarate varabe 2. Decomoston of mutvarate varabe nto bass comonents by bnd sgna searaton methods 3. Identfcaton and emnaton of destructve comonents from bass sgnas 4. Returnng from ceaned bass sgnas to ceaned redcton resuts, n smest case by transformaton nverse to decomoston 5. Choosng the best resuts from ceaned redcton resuts he above methodoogy s resented n neura networ redcton framewor, but t can be addressed to any other regresson modes. 2 Mode resuts ntegraton We assume that after earnng rocess each neura networ resut ncudes two tyes of comonents: constructve assocated wth the target and destructve

2 assocated wth the naccurate earnng data, ndvdua roertes of modes etc. Many of the comonents are common to a the modes due to the same target varabe, earnng data set, smar mode structures or otmzaton methods. ow we eress mode resuts not n terms of nut data but n terms of mentoned above constructve and destructve atent comonents. Let the resut of -th snge outut neura networ, =,..., m, wth observatons, be a functon of constructve mact comonents t, t 2,..., t, and destructve comonents v, v2,..., vq, what gves = F t, t..., t, v, v..., v, ( ( 2 2, q where F s a nage functon what can be treated as mng system ether. For smcty of further anayss we assume near mng system, what gves us redcton resuts as = α... + t α t + β v + β qvq. (2 In cose matr form we have where: = [ t t,..., t ], 2 = α t + β v, (3 t s a matr of target comonents, = [ v v,..., ] v s a q matr of, 2 v q [ α,..., α ], = [ β,..., β q resduas, α = β ] are vectors of coeffcents. In case of many modes we have where = [,..., ] = As, (4, 2 m s a m matr of mode t resuts, s = s a n matr of bass comonents v α β ( n = + q, and A = M M s m n matr of α m β m mng coeffcents. Fgure : System for ntegraton of neura networ resuts A the neura networ resuts taen together gve us one mutvarate varabe whch ncudes atent comonents assocated wth dfferent roertes of artcuar resuts. Our am s to dentfy bass comonents (sgnas and mng matr from observed modes resuts, and reject the destructve comonents v (reacng them by zero to t obtan ŝ =, what gves us urfed target estmaton 0 t ˆ = Asˆ = A. 0 he effectveness of the method hghy deends on the roer dstncton t from v. hs tas requres some decson system, see Fg.. he smest souton s to chec the mact of each bass comonent and ther combnatons on the fna resuts by rejectng them one by one and mng the rest n transformaton nverse to decomoston system. he one of cruca onts n our methodoogy s the roer decomoston choce. 3 Bnd Sgna Searaton and decomoston agorthms From many transformatons whch can be used to obtan bass sgnas we focus on bnd sgnas searaton (BSS. hese methods am at dentfcaton of the unnown sgnas med n the unnown system [7,4,20]. In our case, we are not oong for secfc rea sgnas but rather for nterestng anaytca reresentaton. he BSS methods use dfferent features and roertes of data but most of them can be consdered as oong for data reresentaton of the form (4. o fnd the atent varabes A and s we can use a transformaton defned n m by matr W R, such that (5 y = W, (6 where y s reated to s and satsfes the foowng reaton y = PDs, (7 where P s a ermutaton matr and D s a dagona matr [7,4,7]. he reaton (7 means that estmated sgnas can be rescaed and reordered n comarson to orgna sources. In our case t s not cruca, therefore y can be treated drecty as estmated verson of sources s. here are some dfferent addtona assumtons deendng on artcuar BSS method [7]. We focus on methods based on decorreaton and ndeendent comonent anayss. In further consderaton we assume, for smcty, that n=m. Decorreaton s one of the most ouar statstca rocedures for the emnaton of the deendences estng n the data. In ractce, decorreaton can be erformed by dagonazaton of the correaton matr { ( } R = E ( ( (. (8 = herefore we need such matr W that for transformaton (6 we have R yy = WR W = E, (9

3 where E s any dagona matr. From many methods eadng to the decorreaton matr W (abe the mortant roe ays rnca comonent anayss due to ts nterestng statstca and agebrac roertes [5]. abe : Methods of decorreaton ossbe for modes decomoston Method a Factorsaton Decorreaton Form correaton / 2 / 2 R = R R / 2 W = R LU R = LU W = L Choesy R = G G W = G EIG (PCA R = UΣ U W = U a Defnton and characterstc of the artcuar methods can be found n [0]. Standard decorreaton s very usefu anaytca too but n the contet of the BSS robem t s rather suortng method because of ts sma searaton abtes and t s usuay used as rerocessng before the man agorthms [6,7]. In our case, the most nterestng thng s a romsng anaytca reresentaton of sgnas and therefore, decorreaton can ay an mortant roe n resented methodoogy. Indeendent comonent anayss (ICA s a statstca too, whch aows decomoston of observed varabes nto ndeendent comonents [6,9,7]. yca agorthms for ICA eore hgher order statstca deendences n a dataset, so after ICA decomoston we have got sgnas (varabes wthout any near and non-near statstca deendences [2,6]. hs s the man dfference from the standard correaton methods (e.g. PCA, whch aow us to anayze ony the near deendences. o obtan ndeendent comonents we need to anayze statstca structure of y. he jont robabty of ndeendent varabes can be factorzed by the roduct of the margna robabtes q ( y ( y y y ( y ( y K ( y = ( y, y,..., y. 2 2 n n... n 2 n (0 here are many ways to obtan (0. One of the most ouar methods s based on measure of dfference between y (y and q y (y by the Kubac-Leber dvergence [2,7] D KL y ( y ( y ( y qy ( y = y ( yog dy. ( q ( y + We are oong for a matr W that mnmzes ( so W ot = mn D KL ( y ( W qy ( W. (2 W here are many agorthms for ICA. wo of them are atura Gradent Agorthm [ I E{ f ( y( y ( }] W( W( = µ ( and Fed Pont Agorthm [ E{ f ( y( y ( } dag( E{ f ( y y }] W( y, (3 W( = D.(4 ' Where D = dag( / E{ f ( y y } E{ f ( y }, and f ( y = [ f ( y,..., f ] n ( y n s a vector of nonneartes wth otma form of og( ( y f ( y =. (5 y Otma nonneartes used n (5 need the nowedge of source robabty dstrbutons what s mossbe n genera. here are many roosas to sove ths robem e arametrc and nonarametrc densty estmaton, adatve earnng choce. For unmoda dstrbutons the anayss of urtoss arameters gves us a sme heurstc formua for nonneartes choce [8] tanh( β y r f ( y = sgn( y y y κ 4( y > 0 κ 4( y < 0, (6 κ 4( y = 0 where r >, β are constants and normazed urtoss κ s gven by = κ ( y E{ y }/ E { y } 3. (7 Smooth Comonent Anayss (SmCA s a method of fndng the smooth comonents n a mutvarate varabe wth temora structure [7]. here are varous smoothness measures for BSS e e.g. the Stone s redctabty measure [23], but t s assocated wth the arbtrary choce of movng average form. We roose a new smoothness measure as P ( y = y ( y (, R + δ ( R = 2 (8 where R = ma( y mn( y means range and symbo δ (. means Kronecer deta. he measure P ( y has sme nterretaton: s mama when changes n each ste are equa to range (mama, and s mnma when data are constant. he ossbe vaues are from to 0. o obtan a dfferentabe smoothness measure we can aromate (8 by P ( y = 2 og(cosh( y ( y ( = 2, og(cosh( R + δ ( R (9 where the og(cosh(. functon s used as the aromaton of absoute vaue functon. Our am s to fnd such W = [ w, w 2,..., w3] that for y = W we obtan y = [ y, y2,..., yn ] where y = w mamzes w w = = arg ma( P( w, (20 Havng estmated the frst - smooth comonents the net one s cacuated as most smooth comonents of the resdua

4 = w = arg ma( P ( w ( ww, (2 w = As the numerca agorthm for fndng (2 we can tae ewton method wth mute startng ont run. From many resuts we choose the one wth mama P vaue. In many cases the SmCA gve resuts smar to ICA or other BSS methods but n contradstncton to them t s focused on temora structure of data, what s mortant for robems descrbed by tme seres. 4 Decson system and mutcrtera choce Presented method assumes estence and utzaton of some constructve comonents for each mode resut. he estmaton of the comonents requres choce of the transformaton and abeng of the resutng comonents as the destructve or constructve. hs can be dffcut tas because obtaned comonents mght be not ure constructve or destructve. More recsey, artcuar comonent can have constructve mact on one mode and destructve on the other. Moreover, there may est comonents destructve as a snge but constructve n a grou. herefore, for each transformaton we have to chec the mact of a the comonents subsets on the fna resuts. he smest method for choosng the transformaton and abe ts comonents s as foows. For each transformaton emnate artcuar comonents subsets, combne the remanng ones by mng matr and evauate the fna resuts accordng to chosen crteron. he most sutabe transformaton and abeng woud be the one that ead to the best resuts accordng to the artcuar crteron. he other method can tae advantage of usng many crtera. he mutcrtera aroach s acabe because BSS transformatons gve somethng hysca comonents and the emnaton of the rea hysca nose shoud mrove many crtera. It means utzaton of dfferent tye nformaton e varance, senstvty to outers etc. Whereas the dfferent crtera have dfferent measure characterstcs e scaes, sewness, urtoss etc., any sme aggregaton s not recommended. We roose a decson systems based on Pareto-otmaty of modes resuts n mutdmensona sace of crtera. Let us assume we have a few modes, =,...,m, a set of varous transformatons Φ, =,..., mφ, and some quaty crtera C, =,...,mc. Each transformaton Φ searates some source sgnas S (n the number of m m that can be abeed n 2 -way as constructve or m destructve V ones, S = [, V ], =,..., 2. Let ˆ ( denote the mode resuts based on constructve sgnas obtaned n transformaton Φ, and C ( ( ˆ be the vaue of error measure C for the ˆ. resut ( he choce of the best transformaton Φ and abeng w foow the rue. Let denote the number of Pareto-otma modes resutng from the transformaton Φ and the abeng, where the Pareto-otmaty deends on the crtera C, =,...,m : = card{ 0, 0 = ( ˆ ( ˆ 0 C ( C ( K m 0 C, ( ˆ ( ( ˆ 0 C < C ( = K m 0 C = Km }. C (22 he best constructve sgnas * gve the greatest * * number of Pareto-otma modes ˆ ( *,,...,m =, so: * * * = { * = ma }. (23 * If there are a few best transformatons and abengs * the fna choce s eft to the anayst. It shoud be noted that transformaton choce and comonent dentfcaton can be done on data for whch we have the target, but fna decomoston and mng matr cacuaton shoud contan test data (or redcton resuts when the target s not avaabe., 5 Generazed mng After the destructve comonents are dentfed and emnated the urfed modes resut can be obtaned by ˆ = Asˆ = A[ t 0]. he reacement of the destructve sgnas by zero s equvaent to uttng zero n corresondng to them mng coeffcents. If we eress the mng matr as A = [ a, a2,..., a n ] the urfed resuts can be descrbed as ˆ Asˆ A t 0 Aˆ = = [ ] = [ t v] = As ˆ where A ˆ [ a, a,..., a, 0, 0,..., ] = n (24 Fgure 2: Imrovement rocess based on decomoston he mng matr  s the best matr we can fnd by sme test wth emnatng each combnaton of the comonents. As t was mentoned above the comonents can be not ure, so ther mact shoud have weght other than 0. It means that we can try to fnd the better mng system than descrbed by A (nverse to decomoston system. he new mng system can be formuated even more genera than sme near. We can tae MLP neura networ as the mng system

5 = g B [ g ( B s + b ] +, (25 2( 2 b2 where g (. s a vector of nonneartes, B s a weght matr and b s a bas vector resectvey for -th ayer, =,2. he frst weght ayer w roduce resuts reated to (24 when we tae B Aˆ =. But we can eect that net weghts ayers and nonneartes can mrove resut due to earnng rocess. In other words, n the earnng rocess we search for better mng system startng from system descrbed by  wth nta weghts of B (0 Aˆ =, see Fg.2. 6 Practca eerment In ths aer we anayse the rea robem of forecastng the houry eectrcty consumton for 24 hours ahead basng on the hstorca energy oad and caendar [], [8]. he data reresent eectrcty consumton n Poand durng 4377 days n 80 s and 90 s. he seres from 987 t 996 are randomy dvded nto earnng (2008 days and vadatng set (2007 days and data of 997 (362 days as a testng set. he erformance of the redctons made at 0 a.m. each day s measured by MSE and MAPE crtera [3]. abe 2. he error vaues obtaned by modes mroved by decomoston agorthms. he rates of error reducton are n bracets MAPE[%] MSE [0-4 ] Base a,5523 4,805 ICA,5495 4,349 (0,8% (,09% ICA+,575 3,9783 (2,24% (4,84% PCA,507 3,8929 (3,26% (6,88% PCA+,53 3,9208 (2,64% (6,2% Choesy,5028 3,885 (3,9% (7,07% Choesy+,548 3,9404 (2,42% (5,74% SmCA,530 4,0000 (2,53% (4,32% SmCA+,502 3,9008 (2,7% (6,69% a Best score among the rmary mode resuts: MLP 64 More recse concuson can be drawn from the best scores of the modes before and after decomostons. As we can see the decomoston agorthms enabe mrovement of the resuts by 3% (MAPE to 7% (MSE. If the decomoston agorthm manages to etract ure comonents (PCA and Choesy, the genera mng (neura mode s not better than the sme mng. But n case of neffectve decomoston (ICA we can observe that the genera mng can mrove the resuts. 7 Concusons Fgure 3: Sames of bass sgnas etracted usng PCA. Sgnas y-y3 were dentfed as constructve and y4-y6 as destructve for the mode quaty Seven MLP neura networs wth 60, 62, 64, 66, 68, 70, and 72 neurons n second hdden ayer were traned. he bass sgnas etracton was erformed usng PCA (Fg.3, Choesy and ICA agorthms. he constructve comonents were dentfed usng decson system emoyng Pareto-otmaty condton descrbed n Secton 4. Fgure 4: Rea energy oad and PCA mroved redcton he rea sgna (target and the best mroved modes resuts are shown n Fg.4. We can observe that the redcton systems refects qute we the features of rea rocess. In ths artce we resent a new aroach to the concet of redcton mrovement, where the nformaton from few neura networs s ntegrated. We eore the fact, that there can be many neura networ modes wth dfferent roertes, gvng us dfferent nformaton about redcton robem. Varous crtera can be used for mode resuts evauaton and t s chaengng tas to fnd the sueror mode, because the crtera refect dfferent asects of the robem. On the other hand, even f we have the best mode accordng to our man crteron t can be st vauabe to utze the sghty worse modes for further redcton mrovement. We roose the methodoogy of mode resuts ntegraton based on dentfcaton and emnaton destructve comonents common to many modes. o fnd such comonents we roose BSS methods but the other methods e agebrac decomostons can be used too. Our aroach hes to mrove the modes accuracy and enhances ther generazaton abtes due to combnng the constructve comonents common to a the neura networs. he ractca eerment wth the energy oad redcton confrms vadty of our method. he assumton of modes resuts as a near combnaton of the bass comonents can be etended on nonnear or dynamc systems or even qute omtted n consderatons what eads to machne anayss of many transformatons and sgnas. In such case we have to accet that the

6 transformaton s not assocated wth the mng system of resuts and there s no straghtforward nterretaton of the source sgnas. References [] H. Aae, A new oo at the statstca mode dentfcaton, IEEE ransactons on Automatc Contro, Vo.9, 974, [2] S. Amar, A. Cchoc, and H.H. Yang, A new earnng agorthm for bnd sgna searaton, Advances n eura Informaton Processng Systems IPS-995, MI Press, Cambrdge MA, 996, [3] J.S. Armstrong, F. Cooy, Error Measures For Generazng About Forecastng Methods: Emrca Comarsons, Internatona Journa of Forecastng, Vo.8, 992, [4] C.M. Bsho, eura networs for attern recognton, Oford Unv. Press, Oford UK, 996. [5] L. Breman, Baggng redctors, Machne Learnng, Vo.24, 996, [6] J.F. Cardoso, Hgh-order contrasts for ndeendent comonent anayss, eura Comutaton, Vo., 999, [7] A. Cchoc, S. Amar, Adatve Bnd Sgna and Image Processng, John Wey, Chchester, [8] A. Cchoc, I. Sabaa, S. Cho, B. Orser, and R. Szuu, Sef adatve ndeendent comonent anayss for sub-gaussan and suer-gaussan mtures wth unnown number of sources and addtve nose, Proc.of OLA-97, Vo.2, Hawa USA, 997, [9] P. Comon, Indeendent comonent anayss, a new concet?, Sgna Processng, Vo.36, Esever, 994. [0] G.H. Goub, C.F. Van-Loan, Matr Comutatons, Johns Hons, 3rd ed., 996. [] A. Harvey, S. Kooman, Forecastng Houry Eectrcty Demand Usng me-varyng Snes, Journa of the Amercan Statstca Ass., Vo.88, 993, [2] S. Hayn, eura networs: a comrehensve foundaton, Macman, ew Yor, 994. [3] J. Hoetng, D. Madgan, A. Raftery, and C. Vonsy, Bayesan mode averagng: a tutora, Statstca Scence, Vo.4, 999, [4] A. Hyvärnen, J. Karhunen, E. Oja, Indeendent Comonent Anayss, John Wey, 200. [5]. Joffe, Prnca Comonent Anayss, Srnger- Verag, 986. [6] R.L. Kennedy (ed., Y. Lee, B. Van Roy, C. Reed, and R.P. Lman, Sovng Data Mnng Probems wth Pattern Recognton, Prentce Ha, 997. [7] -W. Lee, M. Groam, A.J. Be, and.j. Sejnows, A Unfyng Informaton-theoretc Framewor for Indeendent Comonent Anayss, Comuters & Mathematcs wth Acatons, Vo.3, 2000,. -2. [8] A. Lendasse, M. Cottre, V. Wertz., M. Verdeysen, Predcton of Eectrc Load usng Kohonen Mas Acaton to the Posh Eectrcty Consumton, Proc. Am. Contro Conf., Anchorage AK, 2002, [9]. Mtche, Machne Learnng, McGraw-H, Boston, 997. [20] R. Szuu, A. Cchoc, Bnd sgna searaton usng second order statstcs, Proc. SPEO 0, 200, [2] R. Szuu, P. Wojewn, and. Zabows, Mode Imrovement by the Statstca Decomoston, Proc. ICAISC 04, Lecture otes n Comuter Scence, Srnger-Verag, Hedeberg, 2004, [22] Y. Yang, Adatve regresson by mng, Journa of Amercan Statstca Assocaton, Vo.96, 200. [23] 5. J.V. Stone, Bnd Source Searaton Usng emora Predctabty, eura Comutaton, 3(7, Juy, 200,