A BRUTE-FORCE ANALYTICAL FORMULATION OF THE INDEPENDENT COMPONENTS ANALYSIS SOLUTION

A BRUTE-FORCE ANALYTICAL FORMULATION OF THE INDEPENDENT COMPONENTS ANALYSIS SOLUTION Dez Erdogmus, Aat Hegde, Keeth E. Hld II, M. Ca Ozturk, Jose C. Prcpe CNEL, Electrcal & Computer Eg. Dept., Uversty of Florda, Gaesvlle, FL 6, USA ABSTRACT May algorthms based o formato theoretc measures ad/or temporal statstcs of the sgals have bee proposed for ICA the lterature. There have also bee aalytcal solutos suggested based o predctve modelg of the sgals. I ths paper, we show that fdg a aalytcal soluto for the ICA problem through solvg a system of olear equatos s possble. We demostrate that ths soluto s robust to decreasg sample sze ad measuremet SNR. Nevertheless, fdg the root of the olear fucto proves to be a challege. Besdes the aalytcal soluto approach, we try fdg the soluto usg a least squares approach wth the derved aalytcal equatos. Mote Carlo smulatos usg the least squares approach are performed to vestgate the effect of sample sze ad measuremet ose o the performace.. INTRODUCTION Idepedet compoets aalyss (ICA) has become a useful tool egeerg ad basc scetfc research. There are may successful algorthms the lterature that fd the depedet compoets of a sgal set. These algorthms mostly explot the depedece assumpto for the sources through formato theoretc measures, hgher order statstcs of the sgals, ad the temporal structures of the sgals through secod order statstcal measures lke cross-correlato at multple lags. Amog the formato theoretc approaches we ca lst Bell ad Seowsk s Ifomax [], Como s mmum mutual formato method [], Yag ad Amar s formato theoretc approaches [], ad Mermad [] by Hld et al. O the hgher order statstcs frot, JADE by Cardoso [5], Pham s decorrelato-of-outputs approach [6], Hyvare s celebrated FastICA [7], Karhue ad Oa s olear PCA approaches [8] are amog the sgfcat ovatos. For ostatoary sources, the tme-varyg cross-correlato of the sgal ca be exploted [9,0,]. A orgal approach, frst proposed by Zbulevsky, s to determe a trasformato such that the represetatos of the sources the trasform space become sparse []. The, the determato of the mxg matrx becomes extremely easy as the observato vectors start lg up alog the matrx colums wth creasg sparsty factor []. Noe of these approaches, however, provde a aalytcal expresso for the mxg matrx or for ts verse, wthout resortg to some drect crtero of optmalty. There have eve bee approaches that utlze the predctve modelg of the sgals to determe a aalytcal soluto for ICA []. Nevertheless, a geerc approach that targets the determato of the mxg matrx wthout restrctve assumptos has ot yet bee proposed. It turs out that the aswer to ths problem les the smplest of all approaches, whch s the subect of ths paper. We called ths approach brute-force ICA, because t reles heavly o forcg out a system of equatos from the data by whch the ukows ca be determed umercally, ad perhaps aalytcally. I fact, whle we were workg o the formulato of the preseted methodology, a paper has appeared that exploted smlar deas to fd a aalytcal soluto for the bld equalzato problem [5]. I ths paper, however, we wll ot focus o the aalytcal soluto. We wll rather cocetrate o the chage the performace of the soluto obtaed through ths brute-force approach whe the umber of trag samples ad the measuremet sgal-to-ose-rato (SNR) are vared.. PROBLEM DESCRIPTION AND SOLUTION Suppose that there exst mutually depedet sgals ( the source vector s) that are mxed by a ukow matrx (called the mxg matrx) H to form the observato vector x accordg to x =Hs. The th etry of the mxg matrx s deoted by h, ad t s the scale factor that multples source s observato x. For smplcty, we cosder the square mxture case, where the umber of observatos s also. I ICA, the task s to determe the depedet source sgals or the mxg matrx havg oly the observed vector samples x(k), where k s the sample (tme) dex. Sce ether H or the source sgals are avalable, the soluto requres some assumptos regardg the statstcs of the source sgals.

Here, we wll resort to the commoly used depedece assumpto. I addto, we wll assume wthout loss of geeralty that the sources are zero-mea ad utvarace. Subtractg the mea of the observato vector from the observed samples satsfes the zero-mea assumpto. O the other had, the ut-varace assumpto s acceptable sce the source ampltude scale factors caot be dstgushed from the mxg matrx etry scale factors. Formally, these two assumptos are expressed as s ] = 0 ad s ] =. I addto, due to the source depedece assumptos, we obta the followg equaltes for teger α. α α s s ] = s ] s ] = 0 () α α α s s ] = s ] s ] = s ] I the last detty of (), s α ] s ether 0 (for α = ), (for α = ), or ukow (for α > ). These two dettes wll prove extremely useful determg the system of equatos for whch we am. Now cosder the secod order ot momets of the observed sgals. x ] = E hl = l = hl ] + hl hm s m ] m= = hl x x ] = E h l h ms m m= = hl h m s m ] = hl h l m= Smlarly, we ca derve expressos for the fourth order ot momets of the observed sgals. Usg the smplfcatos poted out (), these equatos become x ] = hl ] + 6 hl hk k= l+ x x ] = hl h l ] + hl hk h k k= l+ x x ] = hl h l ] + hl h k k= k l + hl hk h l h k k= k l () () Notce () that the fourth order momets of the depedet sources appear as addtoal ukows the equatos. Wth these ew ukows ad the orgal H etres, the total umber of ukows whe the secod ad fourth order ot momets of the observed sgals are cosdered becomes +. I order to determe these ukows, we have extracted +(-)/ equatos from the secod order momets ad +(-)+(-)/ equatos from the fourth order momets, whch amout to. Clearly, we are terested the teger values of. Therefore, for these cases of terest we always have more equatos tha ukows whe the secod ad fourth order ot momets are utlzed ( > +). I the above dscusso, we have ot cosdered the thrd order ot momets, because dog so would troduce equatos that cotaed the thrd order momets of the depedet sources,.e., s ]. For symmetrc source dstrbutos ths momet wll become zero, whch reduces the umber of depedet equatos. Therefore, usg these momets s ot recommeded. Now that we have determed the expressos for the ot momets of the observatos terms of the mxg matrx etres ad the fourth order momets of the sources, the problem reduces to fdg the root of a system of olear equatos of the form f(x)=c, where X s the vector of ukows cosstg of the etres h ad the source momets s ]. The costat vector C, o the other had, cossts of the sample estmates of the secod ad fourth order momets of the observatos,.e., α β x x ], α, β = 0,,,, ad α + β =. I practce, t s recommeded that all equatos be utlzed sce the soluto foud by a overdetermed system of equatos s expected to have smaller fte-sample varace ad more robustess to ose compared to ay soluto that wll be obtaed usg a subset (wth sze + or more) of these equatos. The mportace of the equatos could also be weghted based o the estmated varace of the etres of C due to the fte sample sze. We wll ot however, be cocered wth these ssues ths paper. Clearly, the performace of the soluto obtaed usg ths approach wll be depedet of the sg of the kurtoss of the source sgals. I fact, sce the fourth order momets of the sources are amog the ukows to be determed, the soluto of the algorthm ca be used to determe the values of the kurtoss of the sources. It s possble to wrte out the equatos for the hgher order ot momets of the observed sgals, thus get addtoal or alteratve equatos. However, odd momets are ot desrable due to the same reaso stated earler for the thrd order momets. Hgher order eve momets, o the other had, are less desrable tha the secod ad fourth order momets, smply because as the momet order creases, sample estmates become more

vulerable to outlers ad requre more samples for accurate estmato.. THE SIMPLEST SPECIAL CASE I order to demostrate the level of complexty volved, we wll explctly preset the equatos for a x mxture stuato ths secto. Whe =, the total umber of equatos gve by () ad () s eght. These are gve (). Sce these eght equatos sx ukows do ot coflct wth each other, we ca select a arbtrary subset of sx equatos. x ] x ] xx ] x ] x ] x x ] xx ] x x ] = h + h = h + h = hh + hh = h s ] + h s ] + 6hh = h s ] + h s ] + 6hh = hh s ] + hh s ] + hhh = hh s ] + hh s ] + hhh = hh s ] + hh s ] + hh + hhhh Cosder the selecto of the frst fve equatos ad the last equato. Usg the fourth ad ffth dettes of (), t s possble to determe E [ s ] ad s ] terms of the mxg matrx etres. I addto, usg the frst two dettes, we ca express the dagoal etres of the matrx terms of the off-dagoal etres. After all the substtutos ad takg some combatos of dfferet powers of the thrd ad eghth dettes, we fally obta the followg two equatos that eed to be solved smultaeouy for the dagoal etres of the mxg matrx. 0 = h + [c 0 = h + h + [ hh ( c h)( c h) ] [ h h h ( c h ) h ( ( c 8 [h + h [ c h ( c ( c c ch + c ( c c c h] h ) c ] + ) h h( c 6h( c h)) ) ( c h) (6h( c h ] c h ) ) c5) h( c5 6h( c h )) h) ( c h) (6h( c h) c) ] () (5) I (5), the costats c deote the expectatos o the left had sde of () ad are determed by the data. The frst equato (5) s quadratc, f s cosdered to be a costat. From ths, we obta two solutos for terms of h h h correspodg to dfferet permutatos of the sources. Substtutg oe of these solutos for the secod detty (5) reveals a complcated equato for h. Although we attempted to solve ths equato aalytcally, we were usuccessful. However, t s possble to search for ths root to determe h h h h. The, s determed by the frst detty (5). Moreover, usg the frst two dettes (), t s possble to calculate h ad h. The actual matrx etres ca the be determed by takg the square root of all these values. However, care must be take selectg the sgs of these square roots. These sgs must be cosstet wth all (or oly the selected sx) equatos (). To do ths, oe ca arbtrarly choose the sgs of the dagoal elemets of the soluto. The sgs of the off-dagoal etres ca the be determed usg, for example, the thrd detty (),.e. the c equato.. LEAST-SQUARES APPROACH I the osy fte-sample case, the soluto obtaed by smultaeouy solvg ay subset of the equatos () ( the geeral case () ad ()) mght result suboptmal mxg matrx estmates the least square sese. I order to address ths ssue, t s possble to solve for the matrx etres, as well as the source fourth order momets, by mmzg the followg least squares crtero. J ( X) = ( f ( X) c) G( f ( X) c) T where c s a vector cosstg of the observato ot momets appearg o the left sde of (), X s the vector of ukow parameters cosstg of the mxg matrx etres ad the source fourth order momets, f(x) s the olear fuctos appearg o the rght had sde of (), ad fally G s a postve defte weghtg matrx that could be used to weght the mportace of each equato the soluto. I order to be strctly cosstet wth the least squares theory, ths weghtg matrx could be selected as a dagoal matrx cosstet wth the ftesample estmato varaces of the etres of c. O the other had, computg these estmato varaces s ot a easy task, therefore, oe mght resort to the smple choce of a detty weghtg matrx,.e. G=I. The mmzato ca be carred out usg ay stadard optmzato algorthm. For example, f steepest descet s utlzed, the update algorthm for X becomes (6)

Fgure. SIR (db) hstograms of 50 Mote Carlo smulatos preseted from top to bottom for each of the sample szes 0, 0, 0, 0 5, 0 6. Fgure. SIR (db) hstograms of 50 Mote Carlo smulatos preseted from top to bottom for each of the SNR levels 0dB, 0dB, 0dB, 0dB. X T k+ = Xk c X= X f ( X) η G ( f ( X) ) (7) X Sce the performace surface gve (6) s hghly olear, there wll be local mma that mght trap the algorthm. I order to reach oe of the global optma (there are multple global optma correspodg to dfferet permutatos ad sgs of the separated sources), the soluto offered by (5) could be used as a tal estmate. The least squares procedure the refes the mxg matrx estmate to fd the MSE-optmal soluto. 5. SIMULATION RESULTS I order to study the performace of the brute-force ICA soluto, we have desged two Mote Carlo expermets. I oe of these expermets, we evaluate the performace of the algorthm versus the umber of avalable data k samples. I the secod expermet, we vestgate the robustess of the soluto to measuremet ose by varyg the SNR of the observed sgals. For the smulatos, we use the x case. For every Mote Carlo ru, each etry of the mxg matrx s selected radomly from a uform dstrbuto [-,]. As the performace measure, we utlze the sgal-to-terferece rato (SIR) defed below. Deotg the actual mxg matrx wth H ˆ ad the verse of the estmated mxg matrx wth H, the overall mxg matrx after separato becomes ˆ Q = H H. Lettg q deote the th row of the overall matrx Q, the SIR (db) s defed as SIR = max Q 0 log0 T = q q max Q For each of the sample szes 0, 0, 0, 0 5, ad 0 6, we perform 50 Mote Carlo smulatos usg zeromea, ut-varace sources ad startg from radomly selected matrx estmates. The two source dstrbutos were selected to be uform ad Gaussa. However, otce that the formalsm preseted above does ot mpose ay restrctos o the source dstrbutos other tha depedece, or does ts performace crtcally deped o these dstrbutos. The results of the frst set of Mote Carlo smulatos are preseted Fg.. For each sample sze, the hstogram of the fal SIR values s show a subplot. The subplots are ordered from top to bottom for ascedg sample sze. We clearly observe the expected crease performace as the umber of samples crease from oe hudred to oe mllo. I the secod set of Mote Carlo smulatos, we vary the average SNR at the observed sgals from 0dB to 0dB steps of 0dB. Smlarly, we perform 50 Mote Carlo smulatos usg radomly selected matrx estmates. The sample sze s kept fxed at 0 for all rus. Oce aga, the source dstrbutos are uform ad Gaussa. The SIR measure s ot modfed to accout for the ose the separated sgals, sce ether the algorthm or the demxg structure s tued to reduce ose. Nevertheless, the curret measure gves a dea of how much terferece s comg from the uwated source sgals each output chael. The results of these Mote Carlo smulatos are preseted Fg.. For each SNR level, the hstogram of the fal SIR values s show a subplot. Aga, the subplots are ordered from top to bottom for ascedg SNR. As expected, we observe a crease performace as the ose power drops from beg equal to the sgal power to values eglgble compared to the sgal power. I both sets of smulatos, the steepest descet algorthm sometmes resulted low-qualty solutos exhbtg SIR levels less tha or aroud 0dB. These (8)

Mea SIR N = 0 N = 0 N = 0 Fast ICA 6.7. 0.6 Brute-force 6.9.8 8. Table. Performace of Fast ICA ad the preseted bruteforce ICA approach for the x case wth uform ad Gaussa sources; mea SIR (db) over 50 Mote Carlo smulatos s used as the fgure of mert. separato levels correspod to local mma, therefore they represet suboptmal solutos. The problem of local mma could be avoded by startg adaptato usg a stadard ICA algorthm. After covergece of the stadard algorthm, the soluto could be used as the tal codto for the proposed algorthm order to fe-tue the matrx estmate. I order to demostrate ths, we preset the average performace of Fast ICA [7], a bechmark algorthm, the same expermetal setup (wth a x mxture usg oe uform ad oe Gaussa dstrbuted source). Fast ICA s kow to be very successful the descrbed expermetal settg. For each of 00, 000, ad 0000 sample szes, we have performed 50 Mote Carlo smulatos wth both of these algorthms. The results of brute-force ICA for the same stuato were already preseted Fg. the form of hstograms. The average SIR values obtaed by the solutos gve by these two algorthms for the three trag data szes are lsted Table. Clearly, the preseted brute-force ICA approach s able to acheve a much better separato soluto wth the gve data. Thus, t s possble, for example, to use Fast ICA to obta a suffcetly accurate tal codto for brute-force ICA. The latter approach ca the be mplemeted usg the least-squares methodology descrbed above to obta a more accurate soluto. 6. CONCLUSIONS I the lterature, umerous ICA algorthms are proposed, yet the smplest approach (extractg the equatos for the soluto from the topology usg a brute-force approach o the depedece assumpto) had ot bee tred. I ths paper, we amed to demostrate that t s possble to determe a olear system of equatos from whch the mxg matrx the ICA problem ca be determed. I ths formulato, the propagato of secod ad fourth order momets through the mxg matrx are exploted. As a cosequece, the fourth order momets of the source sgals appeared these equatos as addtoal ukows. Ths way, the determato of a successful soluto has bee made depedet of the sg of the kurtoss of the source sgals. We have attempted to fd the aalytcal soluto for the smplest x stuato. We were able to deduce a sgle olear equato oly oe varable (oe of the dagoal etres of the mxg matrx). However, due to the complexty of ths fal equato, we could ot determe the aalytcal root, whch would yeld the expresso for ths matrx etry. Nevertheless, computer expermets whose results were ot preseted ths paper, sgle dmesoal umercal zero-fdg methods (the stadard fzero fucto of Matlab ) were able to determe ths root very accurately. Oce ths value s determed, the other matrx etres ad the source fourth order momets could be solved aalytcally usg the preseted system of equatos. For geeral practcal purposes, the aalytcal soluto mght ot be feasble due to creasg complexty wth data dmeso. I those stuatos, a least squares approach ca be employed. I ths paper, we have preseted the bascs of such a least squares approach ad we have preseted Mote Carlo smulato results for ths approach. I these smulatos, we have studed the effect of sample sze ad measuremet ose level o the performace of the algorthm. The results showed that the proposed crtero s able to yeld hgh accuracy separato results for sample szes as low as 00. As a future le of research, we wll vestgate the system of equatos that wll arse from utlzg the tmedelayed correlatos of the observed sgals. These mght lead to smpler equatos, thus to a aalytcal expresso for the ICA soluto. Ackowledgmets: Ths work s supported by NSF grat ECS-99009. 7. REFERENCES [] A. Bell, T. Seowsk, A Iformato- Maxmzato Approach to Bld Separato ad Bld Decovoluto, Neural Computato, vol. 7, pp. 9-59, 995. [] P. Como, Idepedet Compoet Aalyss, a New Cocept? Sgal Processg, vol. 6, o., pp. 87-, 99. [] H.H. Yag, S.I. Amar, Adaptve Ole Learg Algorthms for Bld Separato: Maxmum Etropy ad Mmum Mutual Iformato, Neural Computato, vol. 9, pp. 57-8, 997. [] K.E. Hld II, D. Erdogmus, J.C. Prcpe, Bld Source Separato Usg Rey s Mutual Iformato, IEEE Sgal Processg Letters, vol. 8, o. 6, pp. 7-76, 00. [5] J.F. Cardoso, A. Souloumac, Bld Beamformg for No-Gaussa Sgals, IEE Proc. F Radar ad Sgal Processg, vol. 0, o. 6, pp. 6-70, 99.

[6] D.T. Pham, Bld Separato of Istataeous Mxture of Sources va the Gaussa Mutual Iformato Crtero, Sgal Processg, vol. 8, pp. 855-870, 00. [7] A. Hyvare, Fast ad Robust Fxed-Pot Algorthms for Idepedet Compoet Aalyss, IEEE Trasactos o Neural Networks, vol. 0, pp. 66-6, 999. [8] A. Hyvare, J. Karhue, E. Oa, Idepedet Compoet Aalyss, Wley, New York, 00. [9] A. Zehe, K.R. Muller, TDSEP - A Effcet Algorthm for Bld Separato Usg Tme Structure, Proceedgs of ICANN 98, Skovde, Swede, pp. 675-680, 998. [0] L. Parra, C. Spece, Covolutve Bld Separato of No-Statoary Sources, IEEE Trasactos o Speech ad Audo Processg, vol. 8, o., pp. 0-7, 000. [] H.C. Wu, J.C. Prcpe, Geeralzed At-Hebba Learg for Source Separato, Proceedgs of ICASSP 99, Phoex, Arzoa, vol., pp. 07-076, 999. [] M. Zbulevsky, B. Pearlmutter, P. Bofll, P. Kslev, Bld Source Separato by Sparse Decomposto a Sgal Dctoary, Idepedet Compoets Aalyss: Prcples ad Practce, Cambrdge Uversty Press, Cambrdge, Eglad, 000. [] L. Velva, D. Erdogmus, J.C. Prcpe, Uderdetermed Bld Source Separato Usg a Probablstc Source Sparsty Model, Proceedgs of ICA'0, Sa Dego, Calfora, pp. 675-679, 00. [] L. Molgedey, H. Schuster, Separato of Idepedet Sources Usg Tme-Delayed Correlatos, Physcal Revew Letters, vol. 7, pp. 6-67, 99. [5] O. Greller, P. Como, B. Mourra, P. Tebuchet, Aalytcal Bld Chael Idetfcato, IEEE Trasactos o Sgal Processg, vol. 50, o. 9, pp. 96-07, 00.