Is the general form of Renyi s entropy a contrast for source separation?

Size: px

Start display at page:

Download "Is the general form of Renyi s entropy a contrast for source separation?"

Norma Nicholson
5 years ago
Views:

1 Is the geneal fom of Renyi s entoy a contast fo souce seaation? Fédéic Vins 1, Dinh-Tuan Pham 2, and Michel Veleysen 1 1 Machine Leaning Gou Univesité catholique de Louvain Louvain-la-Neuve, Belgium {vins,veleysen}@dice.ucl.ac.be 2 Laboatoie de Modélisation et Calcul Cente National de la Recheche Scientifique Genoble, Fance dinh-tuan.ham@imag.f Abstact. Renyi s entoy-based citeion has been oosed as an objective function fo indeendent comonent analysis because of its elationshi with Shannon s entoy and its comutational advantages in secific cases. These citeia wee suggested based on convincing exeiments. Howeve, thee is no theoetical oof that globally maximizing those functions would lead to seaate the souces; actually, this was imlicitly conjectued. In this ae, the oblem is tackled in a theoetical way; it is shown that globally maximizing the Renyi s entoy-based citeion, in its geneal fom, does not necessaily ovide the exected indeendent signals. The contast function oety of the coesonding citeia simultaneously deend on the value of the Renyi aamete, and on the (unknown) souce densities. 1 Intoduction Blind souce seaation (BSS) aims at ecoveing undelying souce signals fom mixtue of them. Unde mild assumtions, including the mutual indeendence between those souces, it is known fom Comon [1] that finding the linea tansfomation that minimizes a deendence measue between oututs can solve the oblem, u to accetable indeteminacies on the souces. This ocedue is known as Indeendent Comonent Analysis (ICA). This oblem can be mathematically exessed in a vey simle way. Conside the squae, noiseless BSS mixtue model: a K-dimensional vecto of indeendent unknown souces S = [S 1,..., S K ] T is obseved via an instantaneous linea mixtue of them X = AS, X = [X 1,..., X K ] T, whee A is the full-ank squae mixing matix. Many seaation methods ae based on the maximization (o minimization) of a citeion. A secific class of seaation citeia is called contast functions [1]. The contast oety ensues that a given citeion is suitable to achieve BSS. Such a function i) is scale invaiant, ii) only deends on the demixing matix B and of the mixtue densities iii) eaches its global maximum if and only if the tansfe matix W = BA is non-mixing [1]. A matix W is said non-mixing if it belongs to the subgou W of the geneal linea gou GL(K) of degee K, and is defined as: W. = {W GL(K) : P P K, Λ D K, W = PΛ} (1)

2 2 F. Vins, D.-T. Pham and M. Veleysen In the above definition P K and D K esectively denote the gous of emutation matices and of egula diagonal matices of degee K. Many contast functions have been oosed in the liteatue. One of the most known contast function is the oosite of mutual infomation I() [2] whee = BX, which can be equivalently witten as a sum of diffeential entoies h(.): I() =. K K h( i ) h() = h( i ) log det B h(x). (2) The diffeential (Shannon) entoy of X X is defined by h(x). = E[log X ]. (3) Since I() has to be minimized with esect to B, its minimization is equivalent to the following otimization oblem unde a ewhitening ste: max C(B), B SO(K) C(B). = K h(b i X), oblem 1 (4) whee b i denotes the i-th ow of B and SO(K) is the secial othogonal gou SO(K). = {W GL(K) : WW T = I K, det W = +1} with I K the identity matix of degee K. The B SO(K) estiction, yielding log det B = 0, esults fom the fact that, without loss of geneality, the souce can be assumed to be centeed and unit-vaiance (E[SS T ] = I K and A SO(K) if the mixtues ae whitened [8]). Clealy, if W = BA, oblem 1 is equivalent to oblem 2: max C(W), W SO(K). K C(W) = h(w i S). oblem 2 Few yeas ago, it has been suggested to elace Shannon s entoy by Renyi s entoy [4, 5]. Moe ecent woks still focus on that toic (see e.g. [7]). Renyi s entoy is a genealization of Shannon s one in the sense that they shae the same key oeties of infomation measues [10]. The Renyi entoy of index 0 is defined as: h (X) =. 1 1 log X(x)dx, (5) Ω(X) whee 0 and lim 1 h (X) = h 1 (X) = h(x) and Ω(X). = {x : X (x) > 0}. Based on simulation esults, some eseaches have oosed to modify the above BSS contast C(B) defined in oblem 1 by the following modified citeion C (B). = K h ( i ), (6)

3 Is the geneal fom of Renyi s entoy a contast fo souce seaation? 3 assuming imlicitly that the contast oety of C (B) is eseved even fo 1. This is clealy the case fo the secific values = 1 (because obviously C 1 (B) = C(B)) and = 0 (unde mild conditions); this can be easily shown using the Entoy Powe and the Bunn-Minkowski inequalities [3], esectively. Howeve, thee is no fomal oof that the contast oety of C (B) still holds fo othe values of. In ode to check if this oety may be lost in some cases, we estict ouselves to see if a necessay condition ensuing that C (B) is a contast function is met. Moe secifically, the citeion C (W) should admit a local maximum when W W. To see if this condition is fulfilled, a second ode Taylo develoment of C (W) is ovided aound a non-mixing oint W W in the next section. Fo the sake of simlicity, we futhe assume K = 2 and that a ewhitening is efomed so that we shall constaint W SO(2) since it is sufficient fo ou uoses, as shown in the examle of Section 3 (the extension to K 3 is easy). 2 Taylo develoment of Renyi s entoy Setting K = 2, we shall study the vaiation of the citeion C (W) due to a slight deviation of W fom any W W SO(2) of the fom W EW whee E =. [ ] cos θ sin θ (7) sin θ cos θ and θ 0 is a small angle. This kind of udates coves the neighbohood of W SO(K): if W, W SO(2), thee always exists Φ SO(2) such that W = ΦW ; Φ can be witten as E and if W is futhe esticted to be in the neighbohood of W, θ must be small enough. In ode to achieve that aim, let us fist focus on a fist ode exansion of the citeion, to analyse if non-mixing matices ae stationay oints of the citeion. This is a obviously a necessay condition fo C (B) to be a contast function. 2.1 Fist ode exansion: stationaity of non-mixing oints Let Z be a andom vaiable indeendent fom. Fom the definition of Renyi s entoy given in eq. (5), it comes that Renyi s entoy of + ɛz is h ( + ɛz) = 1 1 log +ɛz(x)dx, (8) whee the density +ɛz educes to, u to fist ode in ɛ [9]: +ɛz (y) = (y) ɛ E[(Z = x) (x)] x + o(ɛ). (9) x=y Theefoe, we have: +ɛz(y) = (y) ɛ 1 (y) [E(Z = x) (x)] x + φ(ɛ, y), (10) x=y

4 4 F. Vins, D.-T. Pham and M. Veleysen whee φ(ɛ, y) is o(ɛ). Hence, noting that log(1+a) = a+o(a) as a 0, equations (8) and (9) yield 1 h ( + ɛz) = h ( ) ɛ 1 1 (y)[e(z ) ] + o(ɛ). (11) But, by integation by ats, one gets 1 1 (y)[e(z ) ] = 1 1 (y)e(z = y), (12) yielding 1 (y)[e(z ) ] 1 ɛ (y)e(z = y) = ɛ. 1 (13) Fom the geneal iteated exectation lemma (. 208 of [6]), the ight-hand side of the above equality equals ɛ E[ 2 ( ) ( )Z] = ɛe[ψ ( )Z], (14) if we define the -scoe function ψ ( ) of as ψ ( ) =. 2 ( ) ( ) = 1 ( ) ( ) ( ). (15) Obseve that the 1-scoe educes to the scoe function of, defined as (log ). Then, using eq. (11), noting = EW S, cos θ = 1+o(θ) and sin θ = θ+o(θ), the citeion C (EW ) becomes u to fist ode in θ: C (EW ) = h ( 1 ) h ( 2 ) C { } (W ) ± θ E[ψ (S 1 )S 2 ] E[ψ (S 2 )S 1 ]. (16) The sign of θ in the last equation deends on matix W ; fo examle, if W = I 2, it is negative, and if the ows of I 2 ae emuted in the last definition of W, it is ositive. Remind that the citeion is not sensitive to a left multilication of its agument by a scale and/o emutation matix. Fo instance, C (W ) = C (I 2 ) = h (S 1 ) h (S 2 ). It esults that since indeendence imlies non-linea decoelation, both exectations vanish in eq. (16) and C (W) admits a stationay oint whateve is W W. 1 Povided that thee exist ɛ > 0 and an integable function Φ(y) > 0 such that fo all y R and all ɛ < ɛ, φ(ɛ, y)/ɛ < Φ(y). It can be shown that this is indeed the case unde mild egulaity assumtions.

5 Is the geneal fom of Renyi s entoy a contast fo souce seaation? Second ode exansion: chaacteization of non-mixing oints Let us now chaacteize these stationay oints. To this end, conside the second ode exansion of +ɛz ovided in [9] (Z is assumed to be zeo-mean to simlify the algeba): +ɛz = ɛ2 E(Z 2 ) + o(ɛ 2 ). (17) Theefoe, since Renyi s entoy is not sensitive to tanslation we have, fo > 0: h ( + ɛz) = h ( ) + ɛ2 2 1 (y) va(z) + o(ɛ 2 ), (18) 1 }{{}. =J ( ) whee J ( ) is called the -th ode infomation of. Obseve that the fist ode infomation educes to J 1 ( ) = E[ψ, 2 ], i.e. to Fishe s infomation [2]. In ode to study the natue of the stationay oint eached at W (minimum, maximum, saddle), we shall check the vaiation of C esulting fom the udate W EW u to second ode in θ. Clealy, cos θ = 1 θ 2 /2 + o(θ 2 ) and tan θ = θ + o(θ 2 ), the citeion then becomes: C (EW ) = h ( 1 ) h ( 2 ) = h (S 1 + tan θs 2 ) h (S 2 tan θs 1 ) 2 log cos θ = C (W ) θ2 2 [J (S 1 )va(s 2 ) + J (S 2 )va(s 1 )] 2 log 1 θ2 2 + o(θ2 ) = C (W ) θ2 2 [J (S 1 )va(s 2 ) + J (S 2 )va(s 1 ) 2] + o(θ 2 ) (19) whee we have used H (α ) = H ( ) + log α, fo any eal numbe α > 0. This clealy shows that if the souces shae a same density with vaiance va(s) and -th ode infomation J (S), the sign of C (EW ) C (W ) equals, u to second ode in θ to sign(1 J (S)va(S)). In othe wods, the citeion eaches a local minimum at any W W if J (S i )va(s i ) < 1, instead of an exected global maximum. In this secific case, maximizing the citeion does not yield the seeked souces. 3 Examle A necessay and sufficient condition fo a scale invaiant citeion f(w), W SO(K) to be an othogonal contast function is that the set of its global maximum oints matches the set of the othogonal non-mixing matices, i.e. agmax W SO(K) f(w) = W. Hence, in the secific case whee the two souces shae the same density S, it is necessay that the citeion admits (at least) a local maximum at non-mixing matices. Consequently, accoding to the esults deived in the evious section, the J (S)va(S) < 1 inequality imlies that the souces cannot be ecoveed though the maximization of C (B).

6 6 F. Vins, D.-T. Pham and M. Veleysen 3.1 Theoetical chaacteization of non-mixing stationay oints The above analysis would be useless if the J (S)va(S) < 1 inequality is neve satisfied fo non-gaussian souces. Actually, it can be shown that simle and common non-gaussian densities satisfies this inequality. This is e.g. the case of the tiangula density. We assume that both souces S 1, S 2 shae the same tiangula density 2 T : 1 s/ T (s) =. 6 6 if s 6 (20) 0 othewise. Obseve that E[S i ] = 0, va(s i ) = 1, i {1, 2}, and note that using integation by ats, the -th ode infomation can be ewitten as 2 (y)[ J ( ) = (y)]2 dy. Then, fo S {S 1, S 2 } and noting u =. 1 s/ 6: J (S) = 6 (1 s/ 6) 2 ds (1 s/ = 1 0 u 2 du ( + 1)/[6( 1)] if > 1 6) 0 ds u du = if 1 Thus J (S)va(S) < 1 if and only if ( + 1)/[6( 1)] < 1. But fo 1, the last inequality is equivalent to 0 > ( + 1) 6( 1) = ( 2)( 3). Theefoe J (S)va(S) < 1 if and only if 2 < < 3, as shown in Figue 1(a). We conclude that fo a ai of tiangula souces, the citeion C (B) is not a contast fo 2 < < Simulation Let us note = [ 1, 2 ] T, = W θ S, whee W θ is a 2D otation matix of angle θ of the same fom of E but whee θ can take abitay values [0, π]. The citeion (h ( 1 )+h ( 2 )) is lotted with esect to the tansfe angle θ. Obviously, the set of non-mixing oints educes to W = {W θ : θ {kπ/2 k Z}}. Dawing this figue equies some aoximations, and we ae awae about the fact that it does not constitute a oof of the violation of the contast oety by itself; this oof is ovided in the above theoetical develoment whee it is shown that out of any oblem of e.g. density estimation o exact integation aoximation, Renyi s entoy is not always a contast fo BSS. The uose of this lot is, comlementay to Section 2, to show that in actice, too, the use of Renyi s entoy with abitay value of might be dangeous. Figue 1(b) has been dawn as follows. Fo each angle θ [0, π], the exact tiangula obability density function T is used to comute the df of sin θs 2 This density is iecewise diffeentiable and continuous. Theefoe, even if the density exansions ae not valid eveywhee, eq. (19) is still of use.

7 Is the geneal fom of Renyi s entoy a contast fo souce seaation? = = (a) 2.31 = π/2 π θ (b) Fig. 1. Tiangula souces. (a): log(j (S)va(S)) vs. (b): Estimated Renyi s citeion C (E) vs θ. The citeion is not a contast function fo = 2.5 and = 5. and cos θs, S T, by using the well-known fomula of the df of a tansfomation of andom vaiables. Then, the outut dfs ae obtained by convoluting the indeendent souces scaled by sin θ and cos θ. Finally, Renyi s entoy is comuted by elacing exact integation by Riemannian summation esticted on oints wee the outut density is lage than τ = 10 4 to avoid numeical oblems esulting fom the log oeato. At each ste, it is checked that the dfs of sin θs, cos θs, 1 and 2 integate to one with an eo smalle than τ and that the vaiance of the oututs deviates fom unity with an eo smalle than τ. Note that at non-mixing oints, the exact density T is used as the outut df to avoid numeical oblems. The two last lots of Figue 1(b) clealy indicate that the oblem could be emhasized even when dealing with an aoximated fom of Renyi s entoy. On the to of the figue ( = 1), the citeion C (W) = C(W) (o moe ecisely, C (B) = C(B)) is a contast function, as exected. On the middle lot ( = 2.5), C (W θ ) admits a local minimum oint when W θ W (this esults fom J (S)va(S) < 1), and thus violates a necessay equiement of a contast function. Finally, on the last lot (=5), the citeion is not a contast even though J (S)va(S) > 1 since the set of global maximum oints of the citeion does not coesond to the set W. 4 Conclusion In this ae, the contast oety of a well-known Renyi s entoy based citeion fo blind souce seaation is analyzed. It is oved that at least in one ealistic case, globally maximizing the elated citeion does not ovide the exected souces, whateve is the value of Renyi s exonent; the tansfe matix W globally maximizing the citeion might be a mixing matix, with ossibly moe than one non-zeo element e ow. Even wost, it is not guaanteed that the citeion eaches a local maximum at non-mixing solutions! Actually, the

8 8 F. Vins, D.-T. Pham and M. Veleysen only thing we ae sue is that the citeion is stationay fo non-mixing matices. This is a mee infomation since if the citeion has a local maximum (es. minimum) oint at mixing matices, then a stationay oint might also exist at mixing solution, i.e. at W such that the comonents of WS ae not ootional to distinct souces. Consequently, the value of Renyi s exonent has to be chosen with esect to the souce densities in ode to satisfy K J (S i )va(s i ) > K (again, this is not a sufficient condition: it does not ensue that the local maximum is global). Unfotunately, the oblem is that the souces ae unknown. Hence, nowadays, the only way to guaantee that C (B) is a contast function is to set = 1 (mutual infomation citeion) o = 0 (log-measue of the suots citeion, this equies that the souces ae bounded); it can be shown that counte-examles exist fo any othe value of, including = 2. To conclude, we would like to oint out that contaily to the kutosis citeion case, it seems that it does not exist a simle maing φ[.] (such as e.g. the absolute value o even owes) that would match the set agmax B φ[c (B)] to the set {B : BA W} whee W is the set of non-mixing matices, because thee is no infomation about the sign of the elevant local otima. Refeences 1. P. Comon. Indeendent comonent analysis, a new concet? Signal Pocessing, 36(3): , T. M. Cove and J. A. Thomas. Elements of Infomation Theoy. Wiley Seies in Telecommunications. Wiley and Sons, Inc., A. Dembo, T. M. Cove, and J. A. Thomas. Infomation theoetic inequalities. IEEE Tansactions on Infomation Theoy, 37(6): , D. Edogmus, K.E. Hild, and J. Pincie. Blind souce seaation using enyi s mutual infomation. IEEE Signal Pocessing Lettes, 8(6): , june D. Edogmus, K.E. Hild, and J. Pincie. Blind souce seaation using enyi s α-maginal entoies. Neuocomuting, 49(49):25 38, R. Gay and L. Davisson. An Intoduction to Statistical Signal Pocessing. Cambidge Univesity Pess, K.E. Hild, D. Edogmus, and J. Pincie. An analysis of entoy estimatos fo blind souce seaation. Signal Pocessing, 86: , A. Hyväinen, J. Kahunen, and E. Oja. Indeendent comonent analysis. John Willey and Sons, Inc., New ok, D.-T. Pham. Entoy of a vaiable slightly contaminated with anothe. IEEE Signal Pocessing Lettes, 12(7): , A. Renyi. On measues of entoy and infomation. Selected aes of Alfed Renyi, 2: , 1976.

Analysis of Arithmetic. Analysis of Arithmetic. Analysis of Arithmetic Round-Off Errors. Analysis of Arithmetic. Analysis of Arithmetic

Analysis of Arithmetic. Analysis of Arithmetic. Analysis of Arithmetic Round-Off Errors. Analysis of Arithmetic. Analysis of Arithmetic In the fixed-oint imlementation of a digital filte only the esult of the multilication oeation is quantied The eesentation of a actical multilie with the quantie at its outut is shown below u v Q ^v The