Downloae rom orbit.tu.k on: Dec 06, 2017 Shite Inepenent Component Analysis Mørup, Morten; Masen, Kristoer Hougaar; Hansen, Lars Kai Publishe in: 7th International Conerence on Inepenent Component Analysis an Signal Separation Publication ate: 2007 Document Version Early version, also known as pre-print Link back to DTU Orbit Citation (APA): Mørup, M., Masen, K. H., & Hansen, L. K. (2007). Shite Inepenent Component Analysis. In 7th International Conerence on Inepenent Component Analysis an Signal Separation: ICA2007 (pp. 89-96) General rights Copyright an moral rights or the publications mae accessible in the public portal are retaine by the authors an/or other copyright owners an it is a conition o accessing publications that users recognise an abie by the legal requirements associate with these rights. Users may ownloa an print one copy o any publication rom the public portal or the purpose o private stuy or research. You may not urther istribute the material or use it or any proit-making activity or commercial gain You may reely istribute the URL ientiying the publication in the public portal I you believe that this ocument breaches copyright please contact us proviing etails, an we will remove access to the work immeiately an investigate your claim.
Shite Inepenent Component Analysis Morten Mørup, Kristoer H. Masen, an Lars K. Hansen Technical University o Denmark Inormatics an Mathematical Moelling Richar Petersens Plas, Builing 321 DK-2800 Kgs. Lyngby, Denmark {mm,khm,lkh}@imm.tu.k Abstract. Delaye mixing is a problem o theoretical interest an practical importance, e.g., in speech processing, bio-meical signal analysis an nancial ata moelling. Most previous analyses have been base on moels with integer shits, i.e., shits by a number o samples, an have oten been carrie out using time-omain representation. Here, we explore the act that a shit in the time omain correspons to a multiplication o i! e in the requency omain. Using this property an algorithm in the case o sourcessensors allowing arbitrary mixing an elays is evelope. The algorithm is base on the ollowing steps: 1) Fin a subspace o shite sources. 2) Resolve shit an rotation ambiguity by inormation maximization in the complex omain. The algorithm is proven to correctly ientiy the components o synthetic ata. However, the problem is prune to local minima an iculties arise especially in the presence o large elays an high requency sources. A Matlab implementation can be ownloae rom [1]. 1 Introuction Factor analysis is wiely use to reconstruct latent eects rom mixtures o multiple eects base on the moel n;m = A n; S ;m + E n;m ; (1) where E n;m is aitive noise. However, this ecomposition is not unique since ea = AQ an e S = Q 1 S yiels same approximation as A; S. Consequently, constraints have been impose such as Varimax rotation or Principal Component Analysis (PCA) [2], statistical inepenence o the sources S as in Inepenent Component Analysis (ICA)[3, 4]. A relate strategy is sparse coing where the objective o minimizing the error is combine with a term penalizing the non-sparsity o S [5]. Factor analysis in the setting o ICA is oten illustrate by the so-calle cocktail party problem. Here mixtures o several speakers are recore in several microphones orming the measure signal. The task is to ientiy the sources
S o each original speaker. However, even in an anechoic environment the mixing moel is typically not accurate because o ierent elays in the microphones. Consier two microphones place at istance L an L + h rom a given speaker. Uner normal atmospheric conitions, the spee o soun is approximately c = 344 m/s while a typical sampling rate is s = 22 khz. Then the elay in samples between the two microphones is given by: #samples= sh c such that the elay increases linearly with the ierence in istance. Consequently, a istance o 1 cm gives a elay o 0.6395 samples while h = 1m leas to a elay o 63.95 samples. Harshman an Hong [6] propose a generalization o the actor moels in which the unerlying sources have specic elays when they reach the sensors. The moel is calle shite actor analysis (SFA), an reas n;m = A n; S ;m e n; + E n;m: (2) In real acoustic environments we expect echoes ue to paths that are create by reection o suraces. To account or general elay mixing eects, the ICA moel has been generalize to convolutive mixtures, see e.g., [79] n;m = ; A n;s ;m + E n;m : (3) Here A is a lter that accounts or the presence o each source in the sensors at time elay. The shite actor moel, thus is a special case o the convolutive moel where the lter coecients A n; = A n; i e n; = else A n; = 0. In act shite mixtures are also seen in many other contexts. For instance, astronomy where star motion Doppler eects inuce requency re shits that can be moelle using SFA. Here we will ocus on the elaye source moel. In [6] strong support was oun or the conjecture that the incorporation o shits can strengthen the moel enough to make the parameters ientiable up to scaling an permutation (essential uniqueness). We will emonstrate that this conjecture is not correct when allowing or arbitrary shits. Inee, the moel is, as or regular actor analysis, ambiguous. In [10] an algorithm was propose to estimate the moel. However, the algorithm has the ollowing rawbacks. 1. All potential shits have to be specie in the moel. 2. Exhaustive integer search or the elays is expensive. 3. The moel only accounts or shits by whole samples. 4. The moel is in general not unique. Prior to the work o [6, 10] Bell an Sejnowski [4] sketche how to hanle time elays in networks base on a moel similar to equation 2. This was urther explore in [11]. Although their algorithms erive graients to search or the elays (alleviating the rst two rawbacks above) the moels are still base on pure integer elays. In [12] a ierent moel base on equally mixe sources, i.e. A = 1, orme by moving averages incorporate non-integer elays by signal interpolation. Yereor [13] solve the SFA moel by joint iagonalization o
Fig. 1. Example o activities obtaine (black graph) when summing three components (gray, blue ashe an re ash-otte graphs) each shite to various egrees (given in samples by the colore numbers). Clearly, the resulting activities are heavily impacte by the shits such that a regular instantaneous ICA analysis woul be inaequate. the source cross spectra base on the AC-DC algorithm with non-integer shits or the 2 2 system. This approach was extene to complex signals in [14]. The algorithm is least squares optimal or equal number o sensors an sources. More sensors than sources is not a problem or conventional ICA; we simply reuce imension by variance ecomposition, this proceure is exact or noiseless mixing. Due to the elays projection base imensional reuction will not reprouce the simple single elay structure, but rather lea to a more general convolutive mixture. We will thereore aim at an algorithm or ning a shit invariant subspace. Hence, solve equation 2 by use o the act that a shit in the time omain can be approximate by multiplication by the complex coecients e i! in the requency omain. This alleviates the rst three rawbacks o the SFA algorithm. We will enote this algorithm a Shit Invariant Subspace Analysis (SISA). To urther eal with shit an rotation ambiguities, we impose inepenence in the complex omain base on inormation-maximization (IM) [4]. Hence, we orm an algorithm or ICA with shite sources (SICA). Notice, that algorithms or ICA in the complex omain without shits have previously been erive, see or instance [9, 15] an reerences therein. 2 Metho an Results In the ollowing U will enote a matrix in the time omain, while e U enotes the corresponing matrix in the requency omain. U an e U enotes 3-way arrays in the time an requency omains respectively. Furthermore, U V enotes the irect prouct, i.e. element-wise multiplication. Also,! = 2 1 M such that eu ( ) = U e i2 1 M. Finally, the i th row o a matrix will be enote U i;:. 2.1 Shit Invariant Subspace Analysis (SISA) In the ollowing we will evice an algorithm to n a shit invariant subspace base on the SFA moel. Consier the SFA moel an its requency transorme n;m = A n; S ;m n; + E n;m; e n; = A n; e S; e i2 1 M n; + e En; : (4)
In matrix notation this can be state as Due to Parseval's ientity the ollowing hols C ls = n;m e = e A ( )e S + e E : (5) ke n;m k 2 F = 1 M n; ke En; k 2 F : (6) Thus, minimizing the least square error in the time an requency omain is equivalent. The algorithm will be base on alternatingly solving or A, S an. S upate: Accoring to equation 5, S can be estimate as es = e A ( ) y e : (7) Although, S is upate in the requency omain the upate version has to remain real when taking the inverse FFT. For S to be real value the ollowing has to hol es M +1 = e S ; (8) where enotes complex conjugate. This constraint is enorce by upating the rst bm=2c + 1 elements, i.e. up to the Nyquist requency, while setting the remaining elements accoring to equation 8. A upate: Let e S (n) ; enote the elaye version o the source signal e S; to the n th channel, i.e. e S (n) ; = e S; e i2 1 M n;. Then equation 2 can be restate as n;: = A n;: S (n) + E n;: ; (9) This is the regular actor analysis problem giving the upate A n;: = n;: S (n)y : (10) upate: The least square error or the moel state in equation 5, is given by C ls = 1 M (e A e ( )e S ) H (e A e ( )e S ); (11) where H enotes the conjugate transpose. Dene T ND 1 = vec( ), i.e. the vectorize version o the matrix such that T n+( 1)N = n;. Let urther eq n;; = e A ( ) n; e S ; ; e E = e e A ( )e S : (12) Then the graient o C ls with respect to n; is given as g n+( 1)N = @C ls = @C ls @T n+( 1)N @ = 1 n; M 2!=[ Qn;; e E e n; ] (13)
The Hessian has the ollowing structure H n+( 1)N;n 0 +( 0 1)N = As a result, ( 2 M 2 M P!2 <[ e Qn;; e Q n0 ; 0 ; ] i n 6= n0 ^ 6= 0 P!2 <[ e Qn;; ( e Q n0 ; 0 ; + e E n0 ; )] i n = n0 ^ = 0 can be estimate using the Newton-Raphson metho (14) T T H 1 g; (15) where is a step size parameter that is tune to keep ecreasing the cost unction. Fig. 2. Results obtaine by a shit invariant subspace analysis (SISA). Let panel: the true actors orming a synthetic ata set. To the let, the strength o the mixing A o each source is inicate in gray color scale. In the mile, the three sources are shown an to the right is given the time elays o each source to each channel. Right panel: The estimate actors rom the SISA analysis. Although, all the variance is explaine the ecomposition has not ientie the true unerlying components but an ambiguous mix. Clearly, as or regular actor analysis the SISA is not unique. 2.2 SISA is not unique Accoring to equation 5, the reconstructe signal in the complex omain is given is a rotation, scaling an shit matrix. Assume the inverse o W ( ) is also a rotation, as e e A ( )e S = e A ( ) W ( ) W ( ) 1 e S :Such that W ( ) = W e i2 1 M ^ scaling an shit matrix, i.e. W ( ) 1 = V e i2 1 M. Since W ( ) W ( ) 1 = I, we n 00 1 i2 W ; 00V 0 ;00e M (^ ; 00 + 0 ; 00 ) = 0 or 6= 0 8 1 or = 0 8 (16)
From = 1 we obtain the relation V = W 1. For the remaining requencies this expression can only be vali i ^ 00 + 00 = 0 (iagonal elements) an ^ 00 + 00 0 = k 0 (o iagonal elements) where k 0 enotes an arbitrary constant. The rst relation gives the constraint that ^ = T. The secon relation urther constraints all the elements o the columns o ^ to be equal. Thus the ambiguity is given by W ( ) = [W iag(e i2 1 M b )]. Where b is a vector escribing the shit ambiguity. 2.3 Shite Inepenent Component Analysis (SICA) A common approach to ICA is the maximum likelihoo (ML) metho [16] which correspons to the approach o maximizing inormation propose in [4]. In the ramework o ML a non-gaussian istribution on the sources is assume such that ambiguity can be resolve up to the trivial ambiguities o scale, permutation an source shiting relative to the time elays. Dene, U e = W ( )e S, i.e. the sources at requency when transorme accoring to the rotation an shit ambiguity escribe in the previous section. The ambiguity can be resolve by maximizing the log-likelihoo assuming the (non-gaussian) Laplace istribution p(e U ) / e j U; e j, i.e. Y Y jet( W ( ) )jp( W ( )e S ) (17) p(e S jw; b ) = p(e S jw; b ) = Such that the log-likelihoo as a unction o W an b L(W; b ) = ln j et( W ( ) )j becomes j W ( )e S j (18) By maximizing L(W; b ) W an b is estimate an a new unambiguous S solution oun by e S = W ( )e S. The corresponing mixing an elays can be estimate alternating between the A an upate. We initialize A as A = AW 1 an i; by the maximum cross-correlation between i;: an S ;:. 3 Discussion Traitionally, ICA analysis is base on subspace analysis oten using singular value ecomposition (SVD). The sources are then oun by rotating the vectors spanning the subspace accoring to a measure o inepenence. Similarly, we erive the SISA algorithm to n a shit invariant subspace by alternating least squares. Shit an rotation ambiguities were solve by imposing inepenence on the amplitues o the requency transorm o the sources. While SVD has a close orm solution the SISA algorithm is non-convex. Estimating A an S has a close orm solution or xe values o, S an A. However, is estimate using an iterative metho potentially leaing to many local minima. Furthermore, the problem becomes increasingly icult or high requency sources an large shits
Fig. 3. Result obtaine using the SICA on the ecomposition oun using SISA. By imposing inepenence, e.g., requiring the amplitues in the requency omain to be sparse, the rotation an shit ambiguity inherite in the moel is resolve. Clearly the true unerlying components an their respective mixing are correctly ientie. However, a local minimum has been oun, resulting in errors in the estimation o the elays or the rst component. ue to aitional local minima. In an example we saw this happen: The SICA algorithm aile in correctly ientiying the elays o the rst component; the component with the highest requencies. A multistart strategy was invoke, we choose the best o ten ranom initializations to obtain a goo initial solution or the estimation o the shit invariant subspace. While our algorithm was base on likelihoo maximization, Yereor [13] evelope an algorithm base on joint iagonalization. The present SISA is potentially useul as a preprocessing step or this latter algorithm when estimating less sources than sensors. Current work comprises perormance comparisons with this algorithm (A. Yereor, personal communication). Previous work base on integer shits conjecture the ecomposition to be unique [6]. When using integer shits some shits might perorm better than others ue to a better integer rouning error. Hence, this might be why the integer shits orme seemingly unique solutions. However, as emonstrate in gure 2 the shite actor analysis moel is not in general unique. But, by imposing inepenence unique solutions can be obtaine up to trivial permutation, scaling an specic onset relative to the elays o the sources as emonstrate in gure 3. The shit/elay moel may prove useul or a wie range o ata where ICA alreay has been employe. Furthermore, the extra inormation o elays can be useul or spatial source localization when combine with inormation o position o the sensors. Future work will ocus on implementing aitional constraints such as non-negativity an attempt to improve the ientiability in the presence o many local minima. The current algorithm can be ownloae rom [1].
Reerences 1. Mørup, M., Masen, K.H.: www2.imm.tu.k/pubb/views/publication_etails.php? i=5206. (2007) 2. Kaiser, H.F.: The varimax criterion or analytic rotation in actor analysis. Psychometrica 23 (1958) 187200 3. Comon, P.: Inepenent component analysis, a new concept? Signal Processing 36 (1994) 287314 4. Bell, A.J., Sejnowski, T.J.: An inormation maximization approach to blin source separation an blin econvolution. Neural Computation 7 (1995) 11291159 5. Olshausen, B. A., F.D.: Emergence o simple-cell receptiove el propertises by learning a sparse coe or natural images. Nature 381 (1996) 607609 6. Harshman, R., Hong, S., Luny, M.: Shite actor analysis part i: Moels an properties. Journal o Chemometrics 17 (2003) 363378 7. Attias, H., Schreiner, C.: Blin source separation an econvolution: the ynamic component analysis algorithm. Neural Computation 10(6) (1998) 13731424 8. Parra, L., Spence, C., Vries, B.: Convolutive blin source separation base on multiple ecorrelation. IEEE Workshop on Neural Networks an Signal Processing (1998) 2332 9. Anemuller, J., Sejnowski, T.J., Makeig, S.: Complex inepenent component analysis o requency-omain electroencephalographic ata. Neural Networks 16(9) (2003) 13111323 10. Harshman, R., Hong, S., Luny, M.: Shite actor analysis part ii: Algorithms. Journal o Chemometrics 17 (2003) 379388 11. Torkkola, K.: Blin separation o elaye sources base on inormationmaximization. Acoustics, Speech, an Signal Processing. ICASSP-96 6 (1996) 35093512 12. Emile, B., Comon, P.: Estimation o time elays between unknown colore signals. Signal Processing 68(1) (1998) 93100 13. Yereor, A.: Time-elay estimation in mixtures. ICASSP 5 (2003) 237240 14. Yereor, A.: Blin source separation in the presence o oppler requency shits. ICASSP 5 (2005) 277280 15. Caroso, J.F., Tulay, A.: The maximum likelihoo approach to complex ica. ICASSP (2006) 673676 16. Hyvarinen, A., Karhunen, J., Oja, E.: Inepenent Component Analysis. John Wiley an Sons. (2001)