QUADRATIC DEPENDENCE MEASURE FOR NONLINEAR BLIND SOURCES SEPARATION

Transcription

1 QUADRATI DPNDN MASUR FR NNLINAR BLIND SURS SPARATIN Sophie Achad Dinh Tuan Pham Univ. of Genoble Laboatoy of Modeling and omputation IMAG.N.R.S. B.P. 5X 84 Genoble edex Fance hitian Jutten Laboatoy of Image and Signal INPG 46 avenue Félix Viallet 8 Genoble edex Fance hitian.jutten@inpg.f ABSTRAT Thi wok focue on a quadatic dependence meaue which can be ued fo blind ouce epaation. Afte defining it we how ome link with othe quadatic dependence meaue ued by Feuevege and Roenblatt. e develop a pactical way fo computing thi meaue which lead u to a new olution fo blind ouce epaation in the cae of nonlinea mixtue. It conit in fit etimating the theoetical quadatic meaue then computing it elative gadient finally minimizing it though a gadient decent method. Some example illutate ou method in the pot nonlinea mixtue.. INTRDUTIN Blind ouce epaation (BSS) conit in extacting independent ouce fom thei mixtue without elying on pecific aumption about the mixtue and the ouce ditibution othe than thei independence. Theefoe mot method which have been popoed ae baed on minimizing ome citeion elated to independence. Such citeion often poee the contat popety in the ene that it can be minimized if and only if the output of the epaation ytem ae mutually independent [ 2]. In the context of linea mixtue contat function can be contucted fom cumulant [] o even coelation if lagged coelation ae included [ 4]. Thi i poible becaue of the tong containt of lineaity of the mixtue ince it i well known that the independence between a et of andom vaiable cannot in geneal be infeed fom the fact that ome of thei coelation and cumulant ae zeo. (ne need to conide all of them.) In the nonlinea mixtue poblem it i theefoe of inteet to conide dependence meaue which completely chaacteize independence in the ene that the meaue Thank to the uopean Bli Poject fo funding. can be zeo if and only if independence ha been achieved. f coue uch meaue can be of inteet in the linea mixtue context too. The mutual infomation i a well known and widely ued dependence meaue. It ue in nonlinea BSS ha been intoduced in Taleb and Jutten [5 6] and Babaie-Zadeh [7] among othe. Thi meaue i howeve difficult to etimate a it involve the etimation of entopy which equie denity etimation. Thi can caue evee difficulty fo high dimenional data. Although it i poible to educe a citeion baed on mutual infomation to the one baed only on the maginal entopie thi appoach can lead to lage bia due to bia in denity etimation. Fo thee eaon it could be of inteet to conide othe dependence meaue. Such a meaue i conideed in Muata [8] and in ikon Kankainen and Koivunen [9] which can be taced back to Roenblatt [] and anothe one in Jodan []. The latte meaue doe not eem eay to compute while the fome i nothing but a weighted um of quaed ditance between the joint chaacteitic function and the poduct of the maginal chaacteitic function and i thu much imple. Thi meaue had been ued by Feuevege [2] to contuct a tet of dependence. In thi pape we tudy the above dependence meaue which we call quadatic dependence meaue. e alo make ome extenion by intoducing the ue of cale facto (to make the meaue cale invaiant) and the ue of geneal kenel function. Futhe we deive the gadient of ou citeion in the geneal context of nonlinea BSS poblem and we invetigate the etimation of thi gadient a well a the citeion itelf. The citeion can thu be minimized though a gadient decent and one can contol the tep ize of the algoithm to enue that it deceae at each tep. Then we focu on the paticula cae of the pot nonlinea (PNL) mixtue model. Some imulation eult ae given howing the good pefomance of the algoithm.

2 Section 2 define the quadatic dependence meaue and povide ome intepetation. The next ection applie thi meaue to the nonlinea BSS poblem by poviding a pactical fomula fo the computation of an empiical veion of the citeion and of it gadient. The pecial cae of PNL i developed in geate detail in ection 4 and ome expeimental eult ae povided in the lat ection. 2. TH QUADRATI DPNDN MASUR e fit ecall that a et of ae mutually independent if and only if andom vaiable! fo any et of ummable function of a eal vaiable.... In fact taking a the indicato function of ome meauable et the above popety i nothing but the uual definition of independence. That it alo hold fo abitay ummable function i an eay extenion. Thu we do not need to check the above equation fo all ummable function but only fo a much malle cla of function. A well known example i given by the cla of complex exponential function. Indeed taking to be of the fom "#"$('*)+-/.# fo ome eal numbe. the above equality mean that the joint chaacteitic function of and the poduct of thei maginal chaacteitic function ae equal. But it i well known that thi popety i a neceay and ufficient condition of mutual independence. In geneal thee ae many poibilitie to contuct a et of tet function which can chaacteize independence. A geneal method of contuction i given in the following Lemma. Lemma 2. Let be a ummable kenel function with Fouie tanfom diffeent fom zeo almot eveywhee. Then the andom vaiable ae independent if and only if -#2 45 # 4! 76 # # To pove thi eult one imply emak that the Fouie tanfom of a convolution poduct of function i a poduct of the Fouie tanfom of the function. Theefoe the above equality i equivalent to 8:9 ;. ;.. B 8:<*=4@?@?@?@ 8:9 /. ;. 8: fo all.. whee denote the Fouie tanfom of and and 8 9 denote the joint and maginal chaacteitic 8 < function of 8 epectively. Thi yield the deied eult. The above eult lead to the definition of a quadatic meaue of dependence a follow Definition 2. Let be a quae ummable kenel function with Fouie tanfom diffeent fom zeo almot eveywhee. Fo a et of andom vaiable B we define the quadatic meaue of thei (mutual) dependence a DF GIHKJML N N (P*N P*N i a cale facto that i a poitive functional of the di- fo all eal contant whee Q :KR R R"!;S and JML N N D UT N V 2X tibutionof uch that ] U_ ` _ <*^ KY ZT N V X\[ `. It i wothwhile to note that in the above definition we aume that the kenel function i quae ummable (and not neceaily ummable). e hall ee late that thi condition i ufficient fo the integal in thi definition to be well defined. Note alo the peence of the cale facto whoe pupoe i to make the meaue cale invaiant (multiplying the andom vaiable by abitay contant doe not change thi meaue). Fom the above Lemma the functional i clealy a dependence meaue which completely chaacteize independence: acb if and only if ae mutually independent. Thi meaue may be elated to thoe intoduced by Feuevege [2] o Roenblatt []. A ituation of inteet i when the kenel i an appoximation to the Diac function. Thu take d":fe -dghgih whee h i a (mall) bandwidth kml paamete and j a denity function. It i clea that N g! and l N g! convege epectively to the <*^ joint denity and the poduct <*^ of the maginal denitie of the andom vaiable g g when h tend to zeo. < = <n Theefoe fo mall h appea a the quadatic ditance between an appoximate moothed veion of the joint denity function and the poduct of the maginal denity function of the andom vaiable g o p g. But a it i hown above i alway a< = dependence <n meaue whethe h i mall o not. In denity etimation the choice of the bandwidth paamete h mut ealize a compomie between bia and vaiance (which i not eay). Hee we may chooe h lage ince we need not bothe vey much about bia a we do not eally need to be exactly a quadatic ditance between the denitie. ne can alo expe the meaue in tem of the chaacteitic function. Afte applying the Paeval fomula (which tate that the Fouie tanfom coneve the nom in q ) and a change of integation vaiable : + " D. GMH 8:9 G B _ Jut L v _ Pw. + Pw. u ()

3 8 Y whee Q R R R! S and J t L /.. B L ;.. 8: ;. (2) The meaue () ha been conideed by ikon et al. [9] and Feuevege [2] but only in the paticula cae whee i the Gauian kenel and. quation () how that i well defined a oon a the function 8 9 i quae ummable ince the chaacteitic function ae bounded. That jutifie ou peviou aumption on when we define. Thu we have at ou dipoal a whole cla of quadatic meaue depending on the choice of the kenel and alo on the bandwidth h if we chooe the kenel to be a caled kenel of the fom j R ghgh. Let u te that the kenel doe not need to be a denity and h doe not need to be vey mall. Thu we have a lot of degee of feedom in chooing them. Since we do not know how thee choice will affect the pefomance of the method we will have to chooe them in an ad hoc manne.. BLIND SUR SPARATIN VIA TH QUADRATI DPNDN MASUR e now apply the quadatic dependence meaue defined above to the nonlinea BSS poblem. Hee the obeva- ae elated to the independent ouce tion... though the elation S whee K S and i an invetible mapping. Howeve without futhe aumption on the mapping it would be impoible to econtuct the ouce baed on the ole hypothei of thei independence. Theefoe one ha to aume that the belong to a eticted ubcla of invetible mapping. Fo example can be the cla of linea invetible map in which cae we ae in peence of linea mixtue. In the cae of nonlinea mixtue only the PNL mixtue will be invetigated detail. u goal i to find a mapping in o that the component of Q ae a independent a poible accoding to the quadatic meaue. Howeve i a theoetical meaue which depend on the unknown ditibution of the data. Theefoe two method can be invetigated :. eplace by an etimate and then poceed to minimize the etimated citeion intead which would (geneally) equie to compute it gadient. 2. poceed with the theoetical citeion compute it gadient to obtain a ytem of equation and only then eplace the unknown gadient by ome etimato to get an etimating ytem of equation. + e pefe the fit appoach a it povide u a mean to contol the deceae of the (empiical) citeion at each tep of the algoithm. Thu it eult in a moe obut algoithm in tem of convegence popetie. In the equel the obeved data will be denoted by o being the ample ize. The coeponding ample of ae then o the -th component of Q o o... timation of Let u emak that the dependence meaue involve only the expectation opeato. Thu a natual etimato of can be obtained by jut eplacing thi opeato with the ample aveage defined a og whee i any function of the data. Thu an etimate of will be whee B J L N N B T N V GIH JIL -N N Ö P*N P*N 2X a T N V X [ and i an empiical veion of depending only on the < o A. Howeve the above expeion of i not uitable fo computation becaue it equie multiple integation. Fotunately thee integation can be avoided by the ue of an altenative fomula. Define a new kenel a d" -di P. Then afte expanding the quae of JML and intechanging the ummation and integation : D G L Q! whee L $ N T : G N o T < ^ X N V "o X #" The fit emak i that one can get a imila fomula a above fo the theoetical meaue. e do not need it o we do not wite it down. The econd emak i that depend on only indiectly though theefoe we can chooe diectly without eve conideing. But cannot be abitay. Fom it definition mut be chooen

4 + + X L uch that it Fouie tanfom i a poitive ummable even function ince it Fouie tanfom coepond to _ 8B9 whee i the Fouie tanfom of a eal quae ummable function. 8:9 Some poible choice fo ae: ( denoting the Fouie tanfom of ). The Gauian kenel : #" ;. D 2. The quae auchy kenel: #" g2 # /. D _._. The negative of the econd deivative of the quae auchy kenel : #" G b# *g # ;. D. _._. The fit two kenel coepond to denity kenel (afte nomalizing) and diffe only by thei tail behavio. But the lat kenel doe not and can be negative. ne may note that the kenel i elated to Mece kenel which ae ued epecially in Suppot Vecto Machine []..2. Gadient of the empiical citeion To minimize the citeion with epect to the epaating tanfomation we need to compute it gadient. Actually it i moe convenient to wok with elative gadient. Theefoe we conide a mall elative incement of namely whee i a mall tanfomation. Thi would induce a mall incement Q of Q and Q of. Since depend only on the nomalized vaiable g we fit compute the vaiation of *g and then apply the eult to deive the change in thi citeion. The change of the nomalized vaiable *g induced by a mall change Q in i Q og * * whee -Ni -NiB N Q! and "! od < A "o "o In pactice i often the tandad deviation of then -N N g. The change of : coeponding to a mall elative incement # of i theefoe up to the fit ode k $ Q Q! whee $ $B L $ : N -N* (' <") -N - (' <") T N X ('.-/ < ) _ with L $BD T (' T N o N*o X o <") X being the -th component of the gadient of L and -N B N V o T X being the deivative of. Finally the elative gadient of map : $ Q o! o $ whee Q od $ Q o!4 $ Q! i given by the linea 4 o! Note that imila calculation can be applied *2 to obtain the elative gadient of the theoetical meaue $. ne get the ame fomula except that and ae eplaced by "! and thei theoetical expeion $ $B L -N N : -N * N*- (' <") -N * k -! (' <") N V T - -/ (' <") whee L (5* l 768D g! i it patial deivative with epect to the -th <") component and.6 D 76 g! and i it deivative. $ A :9 tend to $ and it i< clea A that $ vanihe when the vaiable ae independent. Thee- foe the etimating equation obtained by etting the elative gadient of to zeo ae atified in the limit. Anothe emak concen a popety of the elative gadient aiing fom the invaiance with epect to tanlation of and : one ha $ D b and $ 4. APPLIATIN T PNL MIXTURS Q!-B b. 4.. The poblem Let u focu on the PNL mixtue in which cae the et conit of all mapping of the fom 7; ; D <+ = ";p. = ";p -@? /

5 X # G whee = ; ae the element of an invetible matix and ae monotonou function. The epaating ytem conit of a mapping in of the fom <+ -#2 -# # D -@?. / -# whee ; ae the element of an invetible matix and ae monotonou function. In a non paametic appoach only the aumption of the monotony of the nonlinea mapping i made. But thi i geneally not enough ince ome moothne mut be impoed on thee mapping unle thi i done implicitly though the etimating equation. In a paametic o emipaametic appoach the mapping ae paametized by a vecto paamete with potentially lage dimenion in the latte cae hence they ae inheently moothed. In fact in ou emi-paametic appoach the mapping ae epeented by piecewie linea continuou function o that the vecto paamete conit of the lope of thee function in each piece. In the equel we note o o o that ob o Non paametic appoach The paticula fom of and lead u to expe the mall elative incement of in tem of a mall elative change of and of whee i a mall matix and ae mall function. The expeion of in tem of and can be een to be up to the fit ode $BD N ( N V whee denote the -th component of (the vecto) and ae element of the matix. e take the cale function to be the tandad deviation. The expeion fo the elative gadient can then be obtained in thi non paametic appoach a follow * g "' " # $ Q " 8 V whee I k $ Q! I I ( V! $ and $BD $ $B: g p-n Howeve the popety of the citeion to be invaiant with epect to tanlation how that it depend only on the pacing between ucceive value of the odeed tatitic of the ample auming without lo of geneality that ae inceaing function. onequently the gadient depend only on the change in pacing: e a o K. Thi emak lead to wok only with the deivative of the function which yield ome containt on the moothne. xplicitly with ome calculation the elative gadient can be ewitten a the linea map $ Q. " e a 4.. Paametic and emi-paametic appoach Thee method conit in paameteizing each with a vecto paamete. The expeion of the elative gadient i now $ Q 7 " j o whee j od#"! o$" 2u and " being the odinay gadient of with! epect! to!. e have developed two diffeent paameteization. The fit one conit in appoximating the nonlinea tanfomation with piecewie linea function which yield the method called emi-paametic. The econd one conit in uing an appoximation baed on the quantile function. Thee two method ae not explained hee due to lack of pace. e peent only an expeimental eult uing them. 5. AN XPRIMNTAL RSULT e peent on figue the eult of a imulation of the emi-paametic method with: a gid with bin among the obevation. -# K I G b# g2 # pat.). T -#D # b (given at the end of -# '(*)++(*#" Ub # and Hee we peent only the eult of one imulation. e have alo pefomed othe imulation which have indicated that the pefomance of the algoithm depend on the linea mixtue and that the choice of the kenel i vey impotant. In paticula kenel with heavy tail eem to yield bette eult. Hee we peent an example uing a kenel witten above with heavy tail which i not a denity. In futue wok we will attempt to elate ome popetie of thee diffeent kenel to the pefomance of the algoithm. The

6 < G coefficient and ` which contol the gadient decent in ou expeiment ae fixed empiically : b and `. n figue the top gaph epeent the ditibution of the obevation and the middle one the ditibution of the econtucted ouce. Finally the bottom gaph epeent the compenation of the nonlinea function bevation Y=B*Z Ditibution of at convegence ompoition of non linea funtion at convegence. Fig.. Simulation 6. NLUSIN In thi pape a quadatic meaue i ued fo blind ouce epaation. It etimation baed on the ample aveage and a computation tick lead to compute it gadient without equiing numeical integation. Then it minimization i achieved though a gadient decent method. The algoithm ha been implemented fo PNL mixtue and teted in diffeent ituation. But we only peent in thi pape a imple imulation. Futhe pepective of thi wok conit of chooing the hape of the kenel o a to impove the pefomance of the algoithm. 7. RFRNS [] P. omon Independent component analyi a new concept? Signal Poceing vol. no. 6 pp Ap [2] D.-T. Pham Blind epaation of intantaneou mixtue of ouce via an independent component analyi I Tanaction on Signal Poceing vol. 44 no. pp Nov [] D.-T. Pham Blind epaation of intantaneou mixtue of ouce via the gauian mutual infomation citeion Signal Poceing vol. 8 pp [4] A. Ziehe M. Kawanabe S. Hameling and K.R. Mülle Sepaation of pot-nonlinea mixtue uing ace and tempoal decoelation Poceeding IA 2 San Diego Dec. 2. [5] A. Taleb and. Jutten Souce epaation in potnonlinea mixtue I Tanaction on Signal Poceing vol. no. 47 pp ct [6] A. Taleb Sépaation de Souce dan le Mélange Non Linéaie Ph.D. thei I.N.P.G. - Laboatoie L.I.S [7] M. Babaie-Zadeh n blind ouce epaation in convolutive and nonlinea mixtue Ph.D. thei I.N.P.G. - Laboatoie L.I.S. 22. [8] N. Muata Popetie of the empiical chaacteitic function and it application to teting fo independence Poceeding IA 2 San Diego Dec. 2. [9] J. ikon A. Kankainen and V. Koivunen Novel chaacteitic function baed citeia fo ica Poceeding IA 2 San Diego Dec. 2. [] M. Roenblatt A quadatic meaue of deviation of two-dimenional denity etimate and a tet of independence The Annal of Statitic vol. no. pp [] F.R. Bach and M. I. Jodan Kenel independent component analyi Jounal of Machine Leaning Reeach vol. pp. 48 Jul. 22. [2] A. Feuevege A conitent tet fo bivaiate dependence Intenatinal Statitical Review vol. 6 no. pp [] V. Vapnik The natue of tatitical Leaning Theoy Spinge 998.