INDEPENDENT COMPONENT ANALYSIS USING JAYNES MAXIMUM ENTROPY PRINCIPLE. Deniz Erdogmus, Kenneth E. Hild II, Yadunandana N. Rao, Jose C.

Transcription

1 IDEPEDE COMPOE AALYSIS USIG JAYES MAXIMUM EROPY PRICIPLE Deiz Erdgus, Keeth E. Hild II, Yaduadaa. Ra, Jse C. Pricipe CEL, ECE Departet, Uiversity f Flrida, Gaiesville, FL 326, USA ABSRAC ICA deals with fidig liear prjectis i the iput space alg which the data shws st idepedece. herefre, utual ifrati betwee the prjected utputs, which are usually called the separated utputs due t lis with blid surce separati (BSS, is csidered t be a atural criteri fr ICA. Miiizati f the utual ifrati requires priarily the estiati f this quatity fr the saples, ad the adaptati f the separati atrix paraeters usig a suitable ptiizati apprach. I this paper, we preset a uerical prcedure t estiate a upper bud fr the utual ifrati based desity estiates tivated by Jayes axiu etrpy priciple. he gradiet f the utual ifrati with respect t the adaptive paraeters, the turs ut t be extreely siple.. IRODUCIO Idepedet cpets aalysis (ICA is the prble f fidig directis i the data space such that the idepedece f the prjectis is axiized []. It is a special case f the re geeral prble f blid surce separati (BSS, which deals with btaiig estiates f the uw surce sigals usig a uber f ixed bservatis, where the ixig prcess is als uw [2]. I rder t be able t slve this prble, se assuptis regardig the statistical prperties f the surces ust be ade. Oe cly used assupti is the idepedece f the rigial surces. I the istataeus liear ixture case f BSS, which is idetical t the ICA prble, the relatiship betwee the bservati vectr x ad the surce vectr s is x Hs ( I this expressi, H represets the uw ixig atrix. Liitig urselves t the square ixture case where the uber f bservatis equals the uber f surces, the square atrix H is assued t be ivertible. he separati is achieved by fidig a atrix A that ptiizes se criteri that easures the idepedece f the utputs, which are give by yax. A atural criteri fr easurig idepedece is Sha s utual ifrati [3]. Fr rad variables Y,,Y, whse jit pdf is f Y (y ad argial pdfs are f (y,, f (y, respectively, this utual ifrati is defied as fllws [4]. I( Y f Y ( y lg f y Y ( f ( y dy (2 where y[y,,y ]. Equivaletly, Sha s utual ifrati ca be writte i ters f the argial ad jit etrpies [4]. I( Y H ( Y H ( Y where Sha s etrpy is defied as H ( Y f ( y lg f ( y dy H ( Y f ( y lg f ( y dy Y Y fr argial ad jit etrpies, respectively. Althugh iiizati f the utput utual ifrati is csidered t be a atural criteri, due t difficulties assciated with estiatig it i a rbust aer, tw f the st ppular ICA algriths, aely Ifax [5] ad Fast ICA [6] use ther criteria. Specifically, Ifax axiizes the jit utput etrpy f the liearly trasfred data ad Fast ICA uses furth-rder statistics. he difficulty i usig ifrati theretic easures lies i estiatig the desity f the uderlyig data. Successful algriths ust use rbust prbability desity fucti (pdf estiatrs i rder t iprve cvergece t asypttic perfrace ad t iiize sesitivity t utliers. A c apprach is plyial expasis [7,8,9]. rucati f these expasis itrduce iheret errrs i the estiati f the ifrati theretic criteria. Alterative desity estiati appraches iclude Parze widwig [0], ad rthral basis fuctis []. Kerel estiates are used by several researchers i ICA [2,3,4]. I this paper, we will udertae the iiu utual ifrati apprach i traiig the whiteig-rtati (3 (4 (5

2 tplgy as previusly de by ay ther researchers [7,8,2]. he estiate f utual ifrati, hwever, will be based axiu-etrpy desity estiates btaied i accrdace with Jayes axiu etrpy priciple [5]. his priciple basically states that the prbability desity that best fits the already available experietal data, yet that aes iial citet regardig ay usee easureets shuld be adpted. herefre, the data desity is btaied by slvig a cstraied etrpy axiizati prble, where the cstraits guaratee csistecy with available data ad the axiizati f etrpy (ucertaity represets the iiizati f citet t usee data. I this apprach, sice the desity estiates will be based the axiu etrpy priciple, the estiated value f the criteri will cstitute a upper bud fr its actual value. Due t this iiizati f a upper bud (the axiu value the preseted algrith is, i a sese, a iiax apprach. herefre, this criteri ad the assciated learig algrith will be referred t as Miiax ICA. We expect Miiax ICA t exhibit the rbustess prperties f axiu etrpy ethds. 2. HE OPOLOGY AD HE OBJECIVE he algrith will use the c whiteig-rtati schee i which the separati is perfred i tw steps. he whiteig atrix geerates uit-variace ucrrelated sigals, which are the trasfred by a crdiate rtati i rder t axiize idepedece. Assuig the easureets are (ade zer-ea, the whiteig atrix is deteried fr the eigedecpsiti f the easureet cvariace atrix. Specifically, WΛ -/2 Φ, where Λ is the diagal eigevalue atrix ad Φ is the crrespdig rthral eigevectr atrix f the easureet cvariace atrix ΣE[xx ]. he whiteed sigals are btaied by zwx. he crdiate rtati is achieved by ptiizig a rthral atrix R, which is paraeterized usig Gives rtatis [6], usig the utual ifrati criteri. R( Θ R ( θ (6 i j he Gives paraeterizati fr a rtati atrix ivlves the ultiplicati f i-plae rtati atrices as shw i (6. Here, the vectr Θ ctais the Gives agles, i,,- ad j,,. Each f the atrices θ R ( θ are the s-called i-plae rtatis ad are give by a x idetity atrix whse fur etries are dified as fllws: (i,i th, (i,j th, (j,i th, ad (j,j th etries are dified t read cs θ, -si, si θ, ad cs θ, respectively. here are a ttal f (-/2 Gives agles t be ptiized. θ Csiderig the utual ifrati criteri give i (2, ad ticig that fr utputs btaied fr yrz, the jit etrpy reais cstat fr chagig rtatis, the criteri reduces t the su f argial utput etrpies. J (Θ H ( Y he eliiati f the requireet fr estiatig the jit etrpy is a very sigificat gai i ters f algrithic rbustess, sice the estiati f highdiesial desities requires a expetially icreasig uber f saples. 3. HE MAXIMUM EROPY PROBLEM Give a set f saples {x,,x }, the axiu etrpy desity estiate fr X ca be btaied by slvig the fllwig cstraied ptiizati prble. ax H p X lg p X p X subject t E[ f ( X ] α,..., where the cstrait fuctis f (. are selected a priri by the desiger ad the cstats α are calculated usig the give saples with saple ea apprxiati. Usig calculus f variatis ad the Lagrage ultipliers ethd, the sluti t this cstraied axiizati prble is deteried as p X C( λ exp (9 λ f where λ [ λ K λ ] is the Lagrage ultiplier vectr ad C(λ is the ralizati cstat. he Lagrage ultipliers eed t be slved siultaeusly fr the cstraits. I the ctiuus rad variable case, hwever, this is t a easy tas, sice these equatis ivlve expectati itegrals. Usig itegrati by parts, ad usig the assupti that the actual distributi is clse t the axiu etrpy distributi, it is pssible t derive a uch sipler frula t slve fr the Lagrage ultipliers. Csider α i E[ fi ( X ] fi p X (7 (8 (0 Applyig itegrati by parts with the fllwig defiitis, u px v fi Fi ( du f px dv fi λ

3 where f (. detes the derivative f the cstrait fucti, ad F(. detes its itegral, we btai α i Fi px Fi λ f x p x ( X ( (2 It is iprtat that the cstrait fuctis are carefully selected such that their aalytical itegrals are w. I additi, if p X (x decays faster tha F i (x, the the first ter i (2 ges t zer. his yields α i λ Fi f p X (3 λ E[ Fi ( X f ( X ] λ β i Uder the assupti that the actual ad axiu etrpy distributis are clse t each ther (i the sese that these expectatis are apprxiately equal β i ca be estiated fr the saples f X. Fially, defiig the vectr α [ α K α ] ad the atrix β [β i ], the Lagrage ultipliers are deteried by the fllwig liear syste f equatis: λ β α. 4. MIIMAX ICA LEARIG RULE he iiax ICA apprach is based iiizig the criteri i (7 by adaptig the rtati atrix f the whiteig-rtati schee described i Secti 2. Sice i this paper, we paraeterize the rtati atrix usig Gives agles, the ptiizati algrith will update these paraeters. If the steepest descet apprach is eplyed, the updates bece siple. It ca easily be shw that the derivative f Sha s (argial etrpy f a rad variable, deted by H ( Y, with respect t a Gives agle, deted by θ the pdf f the th utput sigal ad, is give by H ( Y α λ (4 θ I (4, λ is the Lagrage ultiplier assciated with α is the th cstrait fr the pdf f the th utput. he Lagrage ultipliers are btaied easily based the previus discussi. he derivative f the cstrait cstat with respect t the Gives agle is deteried fr the utput saples. Sice by defiiti where f ( y, j j α (5 y, is the j th saple at the th utput fr the j curret agles, the derivative i (4 is α j j I (6, the subscript i y, j f ( y, j y, j R : f ( y, j R : R f ( y, j x j j : R : ad ( R : (6 dete the th rw f the crrespdig atrix. he derivative f the rtati atrix with respect t θ is p q R pj R ( θ R ( θ pj θ i j j p+ (7 R ( θ pj R ( θpj R ( θ θ j q+ i p+ j Oce the gradiet is cstructed fr the idividual derivatives with respect t the Gives agles, these paraeters ca be updated by H ( Y Θt+ Θt η (8 Θ where η is a sall step size. 5. SIMULAIOS I this secti, we destrate the effect f userdeteried paraeters the perfrace f Miiax ICA ad preset cpariss with bechar ICA algriths. All algriths are perated i batch traiig de i all the siulatis. Fr perfrace evaluati, we will use sigal-t-iterferece rati (SIR as the easure, which is ade pssible by the fact that the actual ixig atrix H is w i the siulati eviret. he SIR is defied as [2] 2 ax( O SIR ( db 0 lg0 (9 2 O : O : ax( O where ORWH. his easure is the average rati i decibels (db f the ai sigal pwer i each utput chael t the ttal pwer f the iterferig sigals. I the first set f Mte Carl siulatis, we ivestigate the affect f the uber f saples ad the uber f cstraits the perfrace f Miiax ICA. Fr siplicity, we use a 2x2 istataeus liear ixture case, where the etries f the ixig atrix H is selected radly fr a uifr distributi i [-,] fr each e f the 00 rus. he tw idepedet surce

4 SIR (db distributis are uifr ad Gaussia. Fr cstrait fuctis, we use f (xx, which crrespd t the ets f the rad variable X. Repeatig these experiets fr the uber f cstraits 4,5,6,7,8 ad the uber f saples 00,200,500,750,000, the average SIR surface shw i Fig. is btaied. As expected, regardless f the uber f cstraits beig used, perfrace icreases with a icreasig uber f saples. O the ther had, fr a give uber f saples, the uber f cstraits (ets ca be icreased up t a certai pit t iprve perfrace. After a critical value we expect the perfrace t start decreasig due t the fact that higher rder ets require higher uber f saples fr accurate estiati. I Fig., fr sall uber f saples, we bserve that the perfrace des t icrease whe the rder f ets icrease. he fluctuatis are st liely due t the isufficiet uber f Mte Carl siulatis. If the et rder is larger tha the uber f saples ca tlerate, the perfrace f the algrith wuld degrade. his eas, as re saples are available, the Figure. Average SIR f Miiax ICA versus uber f et cstraits ad uber f saples. SIR(dB C MMI Ifax Miiax Fast ICA Figure 2. Average SIR f Miiax ICA, C s perfrace f Miiax ICA ca be iprved by usig re cstraits. I ur secd case study, we will cduct Mte Carl siulatis t cpare the perfrace f fur algriths: Miiax ICA (with the first fur ets as cstraits, C s iiu utual ifrati (MMI algrith [7], Fast ICA [6], ad Ifax [5]. We perfred se siulatis usig Yag s iiu utual ifrati algrith [8], hwever, these preliiary results were t cparable t the perfrace f the ther algriths. herefre, we d t preset thse results here. I each ru, saples f a surce vectr cpsed f e Gaussia, e Laplacia (super-gaussia, ad e uifrly (sub-gaussia distributed etry were geerated. he 3x3 ixig atrix H, whse etries are radly selected fr the iterval [-,], is als geerated. he ixed sigals are the fed it the fur algriths (fr Fast ICA ad Ifax prewhiteig is applied t speed-up cvergece. he average SIR f all algriths btaied ver 00 Mte Carl rus are shw i Fig. 2. I these siulatis, Ifax was t very successful, because geeric sigid liearities were used at its utput layer that did t atch the exact cdfs f the surces. Althugh the perfrace f Ifax wuld icrease if the liearities were atched t the surce cdfs, that wuld result i a ufair cparis by prvidig additial ifrati t the algrith regardig the statistical structure f the surce sigals. Clearly, the results i Fig. 2 destrate that fr the sae saple set, Miiax ICA achieves better separati. Sice these siulatis used ly 4 ets as cstraits, it is pssible t iprve the perfrace f Miiax ICA sigificatly, especially fr large data sets, by usig additial higher rder ets. Hece, the average SIR curve fr Miiax ICA shw i Fig. 2 culd be regarded as a lwer bud fr its perfrace. 6. COCLUSIOS Idepedet cpet aalysis is a prble where ifrati-theretic ptiizati criteria fids atural applicati. Miiu utput utual ifrati is e such ifrati-theretic criteri, which has bee successfully applied i the ICA ctext. Differet appraches usually fcus eplyig varius pdf estiati techiques, which ca the be substituted i the utual ifrati defiiti t btai a sapleestiate fr this quatity. I this paper, we have ce agai explited this well appreciated ifrati-theretic criteri, alg with the widely accepted whiteig-rtati tplgy, i rder t slve the ICA prble. As fr the pdf estiati, we have utilized the axiu etrpy desity estiates, whse rbustess is supprted by Jayes priciple. We have

5 prpsed a ethd fr deteriig the Lagrage ultipliers f the cstraied etrpy axiizati prble, which leads t the afreetied desity estiates. Sice ICA ivlves ctiuus rad variables (ctiuus-valued sigals, deteriig these paraeters is especially difficult due t the reass discussed i the text. he prpsed apprach fr deteriig the Lagrage ultipliers uerically is based the assupti that certai ets f the actual desity ad the crrespdig axiu etrpy desity are clse t each ther. he sluti the reduces t slvig a syste f liear equatis, fr which these paraeters are deteried. his assupti, i a ICA settig where the sigal desities are t extreely spiy (lie a delta-trai, is usually apprxiately satisfied ad the desity estiates are geerally useful. I a settig with spiy sigal distributis, such as digital cuicatis, where the sigal values are selected fr a fiite alphabet, the assupti wuld bece ivalid. We have destrated with Mte Carl siulatis that, whe the data ets are used as the cstraits, a icrease i the uber f cstraits (i.e., iclusi f higher rder ets prduces a icreased perfrace (if the data legth is sufficiet. I additi, cpariss with three ther bechar algriths (C s MMI, Fast ICA, ad Ifax idicates that the Miiax algrith perfrs well. he cputatial lad f Miiax ICA i batch de is slightly higher tha the ther algriths csidered here. give a rugh idea, usig the first fur ets as cstraits, e update usig Miiax ICA requires the evaluati f saple ets f all the utputs up t rder 8, slvig a 4x4 liear syste f equatis, ad taig the derivative f all utput saples with respect t the Gives agles. Miiax ICA, sice it is based Sha s etrpy, des t require adjustets accrdig t the sub/super-gaussiaity f the surces. It utilizes the preseted traiig data t efficietly extract ifrati fr the give set. Sice the axiu etrpy desity estiate cits iially t usee data, Miiax ICA has gd geeralizati capabilities (as als destrated by the SIR easure used as the figure f erit. Future studies will be directed t ptiizig the cstrait fuctis t axiize perfrace fr a give data set. I additi, a lw-cplexity recursive versi f Miiax ICA fr -lie separati will be pursued. Acwledgets: his wr is partially supprted by SF grat ECS REFERECES [] A. Hyvarie, J. Karhue, E. Oja, Idepedet Cpet Aalysis, Wiley, ew Yr, 200. [2] A. Cichci, S. Aari, Adaptive Blid Sigal ad Iage Prcessig: Learig Algriths ad Applicatis, Wiley, ew Yr, [3] J.F. Cards, A. Suluiac, Blid Beafrig fr -Gaussia Sigals, IEE Prc. F Radar ad Sigal Prcessig, vl. 40,. 6, pp , 993. [4].M. Cver, J.A. has, Eleets f Ifrati hery, Wiley, ew Yr, 99. [5] A. Bell,. Sejwsi, A Ifrati- Maxiizati Apprach t Blid Separati ad Blid Decvluti, eural Cputati, vl. 7, pp , 995. [6] A. Hyvarie, Fast ad Rbust Fixed-Pit Algriths fr Idepedet Cpet Aalysis, IEEE rasactis eural etwrs, vl. 0, pp , 999. [7] P. C, Idepedet Cpet Aalysis, a ew Ccept? Sigal Prcessig, vl. 36,. (3, pp , 994. [8] H.H. Yag, S.I. Aari, Adaptive Olie Learig Algriths fr Blid Separati: Maxiu Etrpy ad Miiu Mutual Ifrati, eural Cputati, vl. 9, pp , 997. [9] D. Erdgus, K.E. Hild II, J.C. Pricipe, Idepedet Cpet Aalysis Usig Reyi s Mutual Ifrati ad Legedre Desity Estiati, Prc. IJC 0, pp , Washigt, DC, 200. [0] E. Parze, O Estiati f a Prbability Desity Fucti ad Mde, Aals f Matheatical Statistics, vl. 33,. 3, pp , 962. [] M. Girlai, Orthgal Series Desity Estiati ad the Kerel Eigevalue Prble, eural Cputati, vl. 4,. 3, pp , [2] K.E. Hild II, D. Erdgus, J.C. Pricipe, Blid Surce Separati Usig Reyi s Mutual Ifrati, IEEE Sigal Prcessig Letters, vl. 8, pp , 200. [3] D. Xu, J.C. Pricipe, J. Fisher, H.C. Wu, A vel Measure fr Idepedet Cpet Aalysis, Prc. ICASSP 98, pp. 6-64, Seattle, Washigt, 998. [4] D.. Pha. Blid Separati f Istataeus Mixture f Surces via the Gaussia Mutual Ifrati Criteri, Sigal Prcessig, vl. 8, pp , 200. [5] E.. Jayes, Ifrati hery ad Statistical Mechaics, Physical Review, vl. 06,. 4, pp , 957. [6] G. Glub, C.V. La, Matrix Cputati, Jh Hpis Uiversity Press, Baltire, MD, 993.