Independent Component Analysis (ICA)

Size: px

Start display at page:

Download "Independent Component Analysis (ICA)"

Imogene Green
5 years ago
Views:

1 SECTION 5 Independent Component Analysis (ICA) ( Independent Component Analysis:Algorithms and Applications Hyvarinen and Oja (2000)) is a variation of Principal Component Analysis (PCA).and a strong competitor to Factor Analysis. ICA is an attempt to decompose complex data into independent subparts (also known as the blind source separation problem or the cocktail party problem.) It attempts to determine the source signals S given only the observed mixtures.x (It is necessary to assume independence of source signals, i.e. the value of one signal does not give any information re other signals). Using singular value decompositon X UDV T and writing S N U and A T DVT we can write N X SA T thus each column of X is a linear combination of the columns of S. Since U is orthogonal, and assuming that the columns of X each have mean zero, this means the columns of S have zero mean, are uncorrelated, and have unit variance. Thus we have X i a ij S j, j1 or (writing X and S as column vectors) X AS AR T RS A S for any orthogonal p p matrix R. p i 1,..., p q NOTE: In Factor Analysis with q p we have X i a ij S j, i, j1 X AS i 1,...,p or where the S are the common factors and represents unique factors. ICA can be viewed as another Factor Analysis rotation method (just like varimax or quartimax); it starts essentially from a Factor Analysis solution and looks for rotations that lead to independent components. ICA assumes the S i are statistically independent (thus determining all the cross moments) rather than uncorrelated.(which determines only the second-order cross moments). Independence implies uncorrelatedness so ICA constrains the estimation procedure to give uncorrelated estimates of the independent components (this reduces the number of free parameters and thus simplifies the problem) The extra moment conditions identify A uniquely. In Factor analysis the S j and i are generally assumed to be Gaussian, and orthogonal transformations AS of Gaussians are still Gaussian. Hence we can estimate the model only up to an orthogonal transformation..thus A is not identifiable for independent Gaussian components. (If just one component is Gaussian, the ICA model can be estimated) However we actually do not want Gaussian source variables (we allow at most one Gaussian source variable) because if S i are Gaussian and the mixing matrix A is orthogonal, the X i will also be Gaussian and uncorrelated with unit variance so the joint density will be completely symmetric and we will have no information on the directions of the columns of the mixing matrix A hence A cannot be estimated. We avoid this identifiability problem by assuming the S i are independent and non-gaussian. so S A 1 X.A T X (because A is orthogonal). We assume X has been whitened (sphered) via SVD to have CovX I.; A is orthogonal and solving the ICA problem means finding an orthogonal S such that the components of S A T X are Mills/Norminton 2007 ICA 1

2 independent and non-gaussian. Writing Y W T X where X AS and setting Z A T W we obtain Y Z T S which is more Gaussian than any of the S i. and least Gaussian when it equals one of the S i i.e. when only one of the elements of Z is nonzero. We want to find W to maximize the nongaussianity of Y; this corresponds (in the transformed coordinate systems) to Z which has only one nonzero component. i.e. Y W T X Z T S is one of the independent components. Finding A to minimize mutual information IA T X requires looking for an orthogonal transformation that gives the most independence between its components. This is equivalent to minimizing the sum of the entropies of separate components of Y, which is equivalent to maximizing their departures from Gaussianity (since Gaussian variables have maximum entropy ). There are two problems: : 1. We cannot determine the variances of the independent components. Since S and A are both unknown. a scalar multiple of one S i can be cancelled out by dividing the corresponding column a i of A by the same scalar. Thus we fix the magnitude of the independent components S i - since they are all random variables, we assume each has unit variance and since they have been centered this means ES i 2 1. Note that we can multiply an independent component by1 without affecting the model so there is also ambiguity of sign. 2. We cannot determine the order of the independent components. Since S and A are both unknowns we are free to change the order of the terms, setting any one of them first. Thus a permutation matrix P and its inverse can be substituted in the model to give X AP 1 PS where AP 1 is the new unknown mixing matrix to be solved for by ICA and the elements of PS are the original independent S i but in different (i.e. permuted) order. Read some required files. drive - D: code.dir - paste(drive, DATA/Data Mining R-Code, sep / ) data.dir - paste(drive, DATA/Data Mining Data, sep / ) source(paste(code.dir, BorderHist.r, sep / )) source(paste(code.dir, WaveIO.r, sep / )) library(fastica) We will create and display two signals (Figure 16) S.1 - sin((1:1000)/20) S.2 - rep((((1:200)-100)/100), 5) S - cbind(s.1, S.2) plot(s.1) plot(s.2) ICA 2 Mills/Norminton 2007

3 Figure 16. Original signals and rotate them a - pi/4 A - matrix(c(cos(a), sin(a), -sin(a), cos(a)), 2, 2) X - S%*%A plot(x[,1]) plot(x[,2]) Figure 17. Rotated signals We combine them and then display them with their histograms. border.hist(s.1, S.2) border.hist(x[,1], X[,2]) Mills/Norminton 2007 ICA 3

4 Figure 18. Border histograms of the original (left) and rotated signals. Now start with the mixed signals and observe what happens to the histograms as we rotate the axes on which the signals are projected. b - pi/36 W - matrix(c(cos(b), -sin(b), sin(b), cos(b)), 2, 2) XX - X for (i in 1:9) { XX - XX%*%W border.hist(xx[,1], XX[,2]) readline( Press Enter... ) 1.5 ICA 4 Mills/Norminton 2007

5 Figure 19. Effect of rotating the projection plane We see that for the fully mixed signals the histograms appear nearly Gaussian. As we move through the different projections the histograms move away from normality. The resulting signals are - plot(xx[,1]) plot(xx[,2]) Figure 20. Result of the ICA Now consider what happens for 3 signals - a sine function, a sawtooth, and a pair of exponentials. S.1 - sin((1:1000)/20) S.2 - rep((((1:200)-100)/100), 5) S.3 - rep(c(exp(seq(0,.99,.01)) , -exp(seq(0,.99,.01)) ), 5) S - cbind(s.1, S.2, S.3) A - matrix(runif(9), 3, 3) X - S%*%A Do an ICA on the mixed data. # Set a random mixing a - fastica(x, 3, alg.typ parallel, fun logcosh, alpha 1, method R, row.norm FALSE, maxit 200, tol , verbose TRUE) Whitening Symmetric FastICA using logcosh approx. to neg-entropy function Iteration 1 tol Iteration 2 tol Mills/Norminton 2007 ICA 5

6 Iteration 3 tol Iteration 4 tol e-06 We then plot the original, mixed, and recovered data. oldpar - par(mfcol c(3, 3), marc(2, 2, 2, 1)) plot(1:1000, S[,1], type l, main Original Signals, xlab, ylab ) for (i in 2:3) { plot(1:1000, S[,i ], type l, xlab, ylab ) plot(1:1000, X[,1 ], type l, main Mixed Signals, xlab, ylab ) for (i in 2:3) { plot(1:1000, X[,i], type l, xlab, ylab ) plot(1:1000, a$s[,1 ], type l, main ICA source estimates, xlab, ylab ) for (i in 2:3) { plot(1:1000, a$s[,i], type l, xlab, ylab ) par(oldpar) Original Signals Mixed Signals ICA source estimates Figure 21. Original, mixed and recovered Repeat the process with four signals. S.1 - sin((1:1000)/20) S.2 - rep((((1:200)-100)/100), 5) s.3 - tan(seq(-pi/2.1,pi/2-.1,.0118)) S.3 - rep(s.3, 4) S.4 - rep(c(exp(seq(0,.99,.01)) , -exp(seq(0,.99,.01)) ), 5) S - cbind(s.1, S.2, S.3, S.4) (A - matrix(runif(16), 4, 4)) [,1] [,2] [,3] [,4] [1,] [2,] [3,] ICA 6 Mills/Norminton 2007

7 [4,] X - S%*%A a - fastica(x, 4, alg.typ parallel, fun logcosh, alpha 1, method R, row.norm FALSE, maxit 200, tol , verbose TRUE) Centering Whitening Symmetric FastICA using logcosh approx. to neg-entropy function Iteration 1 tol Iteration 2 tol Iteration 3 tol Iteration 4 tol Iteration 5 tol e-05 oldpar - par(mfcol c(4, 3), marc(2, 2, 2, 1)) plot(1:1000, S[,1], type l, main Original Signals, xlab, ylab ) for (i in 2:4) { plot(1:1000, S[,i ], type l, xlab, ylab ) plot(1:1000, X[,1 ], type l, main Mixed Signals, xlab, ylab ) for (i in 2:4) { plot(1:1000, X[,i], type l, xlab, ylab ) plot(1:1000, a$s[,1 ], type l, main ICA source estimates, xlab, ylab ) for (i in 2:4) { plot(1:1000, a$s[,i], type l, xlab, ylab ) par(oldpar) Original Signals -5 5 Mixed Signals ICA source estimates Figure 22. For this example we will look at three mixtures of 4 signals (note the warning messages).. A - matrix(runif(12), 4, 3) X - S%*%A a - fastica(x, 4, alg.typ parallel, fun logcosh, alpha 1, Mills/Norminton 2007 ICA 7

8 method R, row.norm FALSE, maxit 200, tol , verbose TRUE) n.comp is too large n.comp set to 3 Centering Whitening Symmetric FastICA using logcosh approx. to neg-entropy function Iteration 1 tol Iteration 2 tol Iteration 3 tol e-05 oldpar - par(mfcol c(4, 3), marc(2, 2, 2, 1)) plot(1:1000, S[,1], type l, main Original Signals, xlab, ylab ) for (i in 2:4) { plot(1:1000, S[,i ], type l, xlab, ylab ) plot(1:1000, X[,1 ], type l, main Mixed Signals, xlab, ylab ) for (i in 2:3) { plot(1:1000, X[,i], type l, xlab, ylab ) plot(0, type n )) # Dummy to fill plot(1:1000, a$s[,1 ], type l, main ICA source estimates, xlab, ylab ) for (i in 2:3) { plot(1:1000, a$s[,i], type l, xlab, ylab ) plot(0, type n ) # Dummy to fill par(oldpar) Original Signals -5 5 Mixed Signals ICA source estimates Figure The next example uses ICA on sounds. This is a demonstration found at the Laboratory of Computer and Information Science (CIS) of the Department of Computer Science and Engineering at Helsinki University of Technology ICA 8 Mills/Norminton 2007

9 For this example we will need to read and write.wav files. A.wav file has the basic structure.described in the next function. read.wavz -z function(d.file)z { zz - file(d.file, rb ) # Open binary file for reading # RIFF chunk RIFF - readchar(zz,4) # Word RIFF (4) file.len - readbin(zz, integer(), 1) # Number of bytes in file (4) WAVE - readchar(zz,4) # Word WAVE (4) # FORMAT chunk fmt - readchar(zz,4) # fmt (3) len.of.format - readbin(zz, integer(), 1) # format length (40 f.one - readbin(zz, integer(), 1, size2) # Number 1 (2) Channel.numbs - readbin(zz, integer(), 1, size2) # Number of channels (2) Sample.Rate - readbin(zz, integer(), 1) # Sample rate (4) Bytes.P.Sec - readbin(zz, integer(), 1) # Bytes/sec (4) Bytes.P.Sample - readbin(zz, integer(), 1, size2) # Bytes/sample (2) Bits.P.Sample - readbin(zz, integer(), 1, size2) # Bits/sample (2) # DATA chunk DATA - readchar(zz,4) # Word DATA (4) data.len - readbin(zz, integer(), 1) # Length of data (4) bias - 2^(Bits.P.Sample - 1) wav.data - rep(0, data.len) # Create a place to store data # Read data based on above parameters wav.data - readbin(zz, integer(), data.len, sizebytes.p.sample, signedf) close(zz) # Close the file wav.data - wav.data - bias # Shift based on bias # Return the information for R list(riffriff, File.Lenfile.len,WAVEWAVE, formatfmt, len.of.formatlen.of.format,f.onef.one,channel.numbschannel.numbs, Sample.RateSample.Rate,Bytes.P.SecBytes.P.Sec, Bytes.P.SampleBytes.P.Sample, Bits.P.SampleBits.P.Sample, DATADATA, data.lendata.len, datawav.data) Set up variables for the data and create the file names for the input, mixed, and output file. numb.source - 9 in.file - matrix(0, numb.source, 1) mix.file - matrix(0, numb.source, 1) out.file - matrix(0, numb.source, 1) for (i in 1:numb.source) { in.file[i,] - paste(data.dir, /source,i,.wav, sep ) mix.file[i,] - paste(data.dir, /m,i,.wav, sep ) out.file[i,] - paste(data.dir, /s,i,.wav, sep ) in.wav - {} for (m in 1:numb.source) { in.wav - c(in.wav, list(read.wav(in.file[m,]))) We can look at the characteristics of the file with wav.char - function(wav) { cat( RIFF, wav$riff, \n ) Mills/Norminton 2007 ICA 9

10 cat( Length, wav$file.len, \n ) cat( Wave, wav$wave, \n ) cat( Format, wav$format, \n ) cat( Format Length, wav$len.of.format, \n ) cat( One, wav$f.one, \n ) cat( Number of Channels, wav$channel.numbs, \n ) cat( Sample Rate, wav$sample.rate, \n ) cat( Bytes/Sec, wav$bytes.p.sec, \n ) cat( Bytes/Sample, wav$bytes.p.sample, \n ) cat( Bits/Sample, wav$bits.p.sample, \n ) cat( Data, wav$data, \n ) cat( Data Length, wav$data.len, \n ) wav.char(in.wav[[1]]) RIFF RIFF Length Wave WAVE Format fmt Format Length 16 One 1 Number of Channels 1 Sample Rate 8000 Bytes/Sec 8000 Bytes/Sample 1 Bits/Sample 8 Data data Data Length Set up a random matrix for mixing A - matrix(runif(numb.source*numb.source),numb.source,numb.source) We will create a matrix (50009) that has one source in each column: mixed - {} for (i in 1:numb.source) { mixed - cbind(mixed, in.wav[[i]]$data) We multiply by the 99mixing matrix to produce a new (5000 9) matrix in which each column is a mixture of the 9 columns of the original matrix. mixed - mixed%*%a We now plot the resulting wave form (Figure 24).old.par - par(mfcol c(numb.source, 1)) par(marc(2,2,2,2)0.1) plot(mixed[,1], type l, main Mixed ) for (m in 2:numb.source) { plot(mixed[,m], type l ) if (dev.cur()[[1]]!1) bringtotop(whichdev.cur()) par(old.par) ICA 10 Mills/Norminton 2007

Figure 24. 9 signals mixed In order to save the signal as a wav file, we need the header information. We cheat a little by simply using the in.

11 Figure signals mixed In order to save the signal as a wav file, we need the header information. We cheat a little by simply using the in.wav header and replacing its data part with the mixed data. The first part of the following code simply creates a mixed list from the in list and the second part does the data replacement. mix.wav - {} for (m in 1:numb.source) { mix.wav - c(mix.wav, list(in.wav[[m]])) for (m in 1:numb.source) { mix.wav[[m]]$data - mixed[,m] write.wav(mix.file[m,], mix.wav[[m]]) # Play them Use the sound library to play the mixed sound. library(sound) play(mix.file[1,]) play(mix.file[2,]) play(mix.file[3,]) play(mix.file[4,]) play(mix.file[5,]) play(mix.file[6,]) play(mix.file[7,]) play(mix.file[8,]) play(mix.file[9,]) # Unmix them We will use fastica to unmix the signals and save then play the results as we did for the mixed signal. Mills/Norminton 2007 ICA 11

12 mixed.all - {} for (i in 1:numb.source) { mixed.all - cbind(mixed.all, mixed[,i]) ICA.wavs - fastica(mixed.all, numb.source, alg.typ parallel, fun logcosh, alpha 1, method R, row.norm FALSE, maxit 200, tol , verbose TRUE) # Save them new.wav - {} for (m in 1:numb.source) { new.wav - c(new.wav, list(in.wav[[m]])) for (m in 1:numb.source) { new.wav[[m]]$data - 5*ICA.wavs$S[,m] write.wav(out.file[m,], new.wav[[m]]) # Play them play(out.file[1,]) play(out.file[2,]) play(out.file[3,]) play(out.file[4,]) play(out.file[5,]) play(out.file[6,]) play(out.file[7,]) play(out.file[8,]) play(out.file[9,]) # Plot them old.par - par(mfcol c(numb.source, 3)) par(marc(2,2,2,2)0.1) plot(in.wav[[1]]$data, type l, main Original ) for (m in 2:numb.source) { plot(in.wav[[m]]$data, type l ) plot(mixed[,1], type l, main Mixed ) for (m in 2:numb.source) { plot(mixed[,m], type l ) plot(ica.wavs$s[,1], type l, ) if (dev.cur()[[1]]!1) bringtotop(whichdev.cur()) for (m in 2:numb.source) { plot(ica.wavs$s[,m], type l ) par(old.par) ICA 12 Mills/Norminton 2007

13 Figure 25. # Original - play play(in.file[1,]) play(in.file[2,]) play(in.file[3,]) play(in.file[4,]) play(in.file[5,]) play(in.file[6,]) play(in.file[7,]) play(in.file[8,]) play(in.file[9,]) Mills/Norminton 2007 ICA 13

Independent Component Analysis (ICA)

Data Mining 160 SECTION 5A-ICA Independent Component Analysis (ICA) Independent Component Analysis (ICA) ( Independent Component Analysis:Algorithms and Applications Hyvarinen and Oja (2000)) is a variation