Independent Component Analysis

Transcription

1 Department of Physics Seminar I b 1st year, 2nd cycle Independent Component Analysis Author: Žiga Zaplotnik Advisor: prof. dr. Simon Širca Ljubljana, April 2014 Abstract In this seminar we present a computational method for blind signal separation. Its purpose is to deduce source signals from a set of various mixed signals without any information about the source signals or mixing process. Independent component analysis is a recently developed method in which the goal is to find source components which are statistically independent, or as independent as possible. We give a description of the basic concepts of this method based on cocktail-party problem example. We also discuss its implementation. Finally, we present some already used applications of independent component analysis.

2 Contents 1 Introduction 2 2 Principles of ICA estimation Definition of ICA model Constraints Estimating ICA model by maximizing non-gaussianity Ambiguities of ICA Applications of ICA Electrocardiography Reflection Cancelling Conclusions 12 References 12 1 Introduction Independent component analysis (ICA) is a statistical computational method for blind signal (source) separation that means separation of source signals (audio, speech, image, geophysical, medical...) from a set of mixed signals. Term 'blind' suggests that there is little or no information about the signals or mixing process. ICA was first introduced in 1986 by Herault and Jutten [1]. Comon provided the most widely used mathematical definition of ICA in 1994 [2]. We will give a more detailed description of it in the next chapter. First fast and efficient ICA algorithm was introduced by Bell and Seynowski in 1995 [3]. From then on many different ICA algorithms were published. One of the most used, including in industrial applications (which we will discuss in the last chapter) is the FastICA algorithm, developed by Hyvärinen and Oja in 1999 [4,5,6]. As the name implies, the basic goal of ICA is to estimate such a transformation of an observed mixture of signals, that the resulting source components are minimally correlated or as independent as possible in a probabilistic sense. Furthermore, the signals have to be non-gaussian. For example, we measure signals ( ), ( ) of two microphones recording the speech of two persons (sources) ( ) and ( ). We assume that each source reaches both microphones at the same time (there is no time/phase delay). That means that the distance between microphones has to be much smaller than the source signal wavelength. However, it is important that the microphones amplify more the signals coming from certain directions (like our ears do), otherwise the measured signals ( ) are all the same. So we can assume that each of these recorded 2

3 signals is a weighted sum (linear combination) of the speech signals. We could express this as a linear equation ( ( ) ( ) ) ( )( ( ) ( ) ) (1) which holds for every timestamp. Here, both sources and matrix elements are unknown. The value of depends on the relative contribution of the source signal to recorded mixture. An illustration of this example is shown in Fig. 1 and Fig. 2. These are, of course, not realistic speech signals, but suffice for the explanation. The original (source) signals are shown in Fig. 1 and the mixed signals could look like those in Fig. 2. The problem is to recover the data in Fig. 1, using only the data in Fig. 2. If we know the parameters, the linear equation (1) is trivial to solve. However, the point is that the parameters are unknown and then the problem is considerably more difficult. Figure 1. Original (source) signals. Lets denote the upper signal ( ), the lower signal ( ). [4] Figure 2. Linearly mixed signals. Here we denote upper mixed signal ( ) and the lower signal ( ). It can be clearly seen, that in ( ), the dominant source signal (the one with larger absolute value of weight) is ( ) from Fig. 1, while in ( ) the dominant source signal is ( ) from Fig. 1. Considering the definition in Eq. (1), we can also obtain that the weight (mixing parameter) is negative. [4] 3

4 The above example is an example of the famous cocktail-party problem, a problem related to cocktail-party effect. The cocktail-party effect is the phenomenon of being able to focus attention on a particular stimulus while filtering out other stimuli. For example one is able to focus on a single conversation in a noisy room. However, this requires hearing with both ears. People with only one functioning ear are much more distracted by interfering noise than people with two healthy ears. The reason is following: the auditory system (sensory system for the sense of hearing) has to be able to localize and process at least two sound sources in order to extract the signals of one sound source out of a mixture of interfering sound sources. Here localizing means turning head into the direction of the source of interest and so amplifying the signal coming from it [7]. A similar condition occurs in ICA a number of obseved sound mixtures must be at least as large as the number of independent sound components. Algortihm alone ofcourse cannot localize (and so amplify) certain signals, therefore only two microphones in a room of three simultaneously speaking people are insufficient in order to separate sound signals of all three speakers. Here, we started a discussion about conditions and limits of ICA. In the next section we describe a generalized cocktail-party problem. In order to be able to solve this problem we also give a detailed insight into the ideas and construction of ICA. 2 Principles of ICA estimation 2.1 Definition of ICA model Here we only describe noiseless linear independent component analysis. Assume that we observe linear mixtures of independent components (sources). Then we can define the ICA model where is the mixing matrix. ICA model (2) describes how the observed data are generated by a process of mixing the components (sources). We have now dropped the time index as (2) holds at any given time instant. With the purpose of efficiently describing ICA, we rather work with random variables instead of the proper time signals from here on. That means we assume each mixture as well as each independent component is a random variable, instead of the amplitude of the signal at certain time. We neglect measurement errors, which could (in case they exist) be added to the right side of (2). We assume that noise is already included in the sources. So there is also no noise at the right side of (2) as we said, we are performing noiseless ICA. We also assume that the signal is centered (random variables have zero mean or ) and that the variables are of unit variance:. Let us now illustrate the ICA model in statistical terms. Consider two independent components, with uniform distributions (2) ( ) { (3) 4

5 The range of values for this uniform distribution were chosen so as to make the mean zero and the variance equal to one, in accordance with the preceding paragraph. Joint probabilty density of and is then uniform on a square, as shown in Fig. 3. Now let us mix these two independent components with the following mixing matrix ( ) (4) The mixing gives us two mixed variables, and. Now, the mixed data has a uniform distribution on a parallelogram (also shown in Fig. 3). But random components and are not independent any more. This can be clearly seen if we are trying to predict the value of one of them, for example, based on the value of the other,. For example, when attains maximum or minimum value, then this completely determines the value of. That means they are not independent. Figure 3. [LEFT] The joint distribution of the independent source components and with uniform distributions. Horizontal axis:, vertical axis:. [RIGHT] The joint distribution of the observed mixtures and. Variables and are not independent any more. For example, when taking maximum or minimum value of, variable is uniquely determined by. Horizontal axis:, vertical axis:. [4] Our main task is to reconstruct sources based on the observed. We have no information about the mixing process by matrix. However, if the matrix is of full rank, a separation or unmixing matrix exists. Then unmixing process can be written as ( ) ( ) (5) Here are the rows of. We attempt to find an estimate of the separation matrix so that we can reconstruct. In order to estimate matrix, several additional requirements (conditions) should be fulfilled. The constraints for the validity and possibility of solving blind signal separation problem with ICA method are discussed in the next section. 5

6 2.2 Constraints Here we describe three important constraints for ICA method to be valid. 1) The problem is well- or over-determined. The number of measured signals has to be equal to or greater than the number of source signals ( ). Otherwise it is impossible to separate the sources using ICA alone. 2) Mixing is instantenous. No delay is permitted between the time each source signal reaches sensors/microphones/observers/recording stations located in different places. In order to avoid (time) delay, recorded signals are aligned using the cross-correlation function (sliding dot product; similar to convolution). For continuous signals and, it is defined as ( )( ) ( ) ( ) (6) If and are functions that differ only by an unknown shift along the -axis, then cross-correlation can be used to find how much must be shifted along the -axis to align with. When the functions match, the value of ( ) is maximized. 3) Statistical independance of sources. The independence of variables is mathematically expressed by the statement that their joint probability density is factorizable in the following way: ( ) ( ) ( ) ( ). Basically the variables are said to be independent if information on the value of does not give any information on the value of, and vice versa. A pair of statistically independent variables ( ) is uncorrelated and has the covariance ( ). Uncorrelatedness is a weaker form of independence: if the variables are independent, they are uncorrelated, but the opposite is not necessary true. The source signals must be statistically independent or as independent as possible. This is difficult to verify in advance because distribution of data is not available in real world problems. 4) Source signals must be non-gaussian (except one signal). (Fig. 4) This is the fundamental restriction of ICA to be possible. Here is an explanation, why more than one Gaussian variable make ICA impossible. Assume that the mixing matrix is orthogonal and the variables and are Gaussian. Then and are Gaussian too and their joint probability density is Gaussian again ( ) ( ) (7) The probabilty density is perfectly symmetric (Fig. 4) and so it does not contain any information on the directions of the columns of the mixing matrix. That means matrix cannot be estimated. One can prove, that the distribution of any orthogonal transformation of the gaussian variables ( ) has exactly the same distribution as ( ) and that and are still independent. Thus, in the case of more than one Gaussian variable, we can only estimate up to an orthogonal transformation and the matrix is not identifiable. But if only one independent component is Gaussian, ICA 6

7 is still possible. Non-Gaussianity is the key to estimating the ICA model, as we will see in the next section. Figure 4. [LEFT] The comparison between probabilty density function of Gaussian and non - Gaussian signal. [RIGHT] The joint multivariate distribution of two independent Gaussian variables (Eq. 6). Horizontal axis:, vertical axis:. 2.3 Estimating ICA model by maximizing non-gaussianity First, we denote one-dimensional projections of the observed signals : where are the vectors to be determined. If one were one of the rows of the generalized inverse (separation matrix) of, this linear combination would actually equal to one of the independent components,. Now the question is how to determine so that it would equal one of the rows in unmixing matrix. We can show from (2) and (8) that this relation holds: 7 (8) (9) That means that is a linear combination of with weights. Now we use a statement that comes from the Central Limit Theorem 'a sum of two independent random variables has a distribution that is closer to Gaussian than a distribution of any of the two original random variables'. In our case, that implies that is more Gaussian than any of the in the linear combination (9). Therefore becomes least Gaussian when it in fact equals one of the independent components (which means that only one of the elements of is nonzero). That means we would take as a vector that minimizes Gaussianity or maximizes non-gaussianity of [4]. In practice, we would start with some vector, compute the direction, in which non-gaussianity is growing most strongly based on the observed mixture vector, and use some gradient method (for example Newton's method) for finding a new vector. The whole algorithm is beyond the scope of this seminar but can be found in [5].

8 To maximize non-gaussianity, first, we must have a measure of non-gaussianity. The classical measure (because of computational simplicity) for non-gaussianity of a continuous random variable is the absoulute value of kurtosis ( ) (10) which is a fourth order cumulant of probability distribution. For Gaussian random variable, the kurtosis is zero. However, practically (when estimated from a measured sample), curtosis is not robust as it is very sensitive to outliers. Its value may strongly depend on only a few observations in the tails of the distribution, which may not be correct at all. The difference between Gaussian distribution and the distribution of could then rather be measured by Shannon (information) entropy ( ) ( ) ( ) (11) where ( ) is the probabilty density function of. The more unpredictable the variable is, the larger its entropy. An important result of information theory is that 'a Gaussian variable has the largest entropy among all random variables of equal variance' [8]. This means entropy could be used as a measure of non-gaussianity. It turns out, that it is more practical to work with negentropy ( ) ( ) ( ) (12) which is zero for a Gaussian variable and positive otherwise. The problem of using negentropy as a measure of non-gaussianity is that ( ) from (11) is not known in advance and that it is computationally very difficult. We therefore use approximations of negentropy, most often ( ) [ ( ) ( ) ] (13) where ( ) is a non-quadratic function of. The following choices of have proved useful [4]: ( ) and ( ) ( ), where. 2.4 Ambiguities of ICA By looking at (2), it is obvious that the following ambiguities will hold (Fig. 5): 1) Variances (energies) of the independent components cannot be determined. Vector of sources and the mixing matrix are both unknown. That means that any scalar multiplier of the source could always be cancelled by dividing the corresponding column of by the same scalar. So the magnitudes (amplitudes) of the independent components are unknown. 2) Sign ambiguity. We could multiply any independent component by (phase reversal) without affecting the model. 3) Order of independent components cannot be determined. The estimated source signals may be recovered in different order. The reason is again that both and are unknown. 8

9 Figure 5. The estimates of the original source signals, shown in Fig. 1. These estimates were made using only the observed signals in Fig. 2. Considering the above mentioned ambiguities of the ICA model, the original signals were very accurately estimated, up to multiplicative signs. [4] 3 Applications of ICA In this chapter, we describe some applications of ICA. The most basic application of ICA, that is the cocktail-party problem as an example of blind source separation, was already explained in the Introduction. There are numerous other practical applications of ICA in different research domains such as: Biomedical signal processing (magnetoencephalography, electroencephalography, electrocardiography). Geophysical signal processing. ICA was applied to seismic signals of tremors recorded at Stromboli volcano [10]. Meteorological signal processing. For example, an ICA based study of a relation between El Nino southern oscillation (ENSO) and Atlantic tropical surface sea temperature anomaly (SST) has been made. [11]. Sonar and radar signal processing. Telecommunications. Separation of the user's own signal from the interfering other users' signals in CDMA (Code-Division Multiple Access) mobile communications. CDMA is used in 3G mobile phones standard [12]. Image processing (image denoising, reflection cancelling, separation of light sources, face recognition, feature extraction). Watermarking [13]. Clustering or Cluster Analysis. Text mining. Recently it has been found that ICA is a powerful tool for analyzing text document data as well, if the text documents are presented in a suitable numerical form. For example the purpose of one study was to find the topics of documents and grouping them accordingly. Other applications where factor analysis and principal component analysis are currently used. We will discuss some of the abovementioned applications in the following subsections. 9

10 3.1 Electrocardiography Routinely recorded electro-cardiograms (ECGs) are often corrupted by different types of artifacts. The term 'artifact' comes from word artificial that means that it is recorded from sources other than the electronic signals of the heart. Examples of artifacts in ECGs are electrical interference from a nearby electrical appliance, patient's muscle tremors as a result of movements, speaking, deep respiration etc. The presence of artifacts in the ECGs is very common and the knowledge of them is necessary to prevent misinterpretation of a heart's rhythm which can lead to wrong diagnoses. Even better is to almost completely remove noise and artefacts from ECGs. This can be done with ICA [14]. Here, an important difficulty is one of the ICA ambiguities determination of the order of the independent components. That means that still a trained physician is needed to manually interpret the deduced independent components. Figure 6. [ABOVE] Electro-cardiogram of a pregnant woman. We can see a weak and noisy signal of the child's heartbeat in signals, and. It is mixed with the signal of the mother's heartbeat and other artefacts. [BELOW] Independent components (sources) of the mixed signals in the picture above. The child's heartbeat can be seen in independent components and. It is faster than mother's heartbeat, which is displayed in all the other ICs, except, which is an artifact, probably generated by respiration during the measurement [9] 10

11 3.2 Reflection Cancelling When we take a picture through a window, the observed image is often a linear superposition of two images: the image of the scene beyond the window and the image of the scene reflected by window. In such cases we cannot view clearly a scene due to reflections from dielectric surfaces (e.g. glass). Since reflections off a semireflecting medium such as a glass window at some angle are partially polarized, the strength of the reflection can be manipulated with a polarizer. However the reflection can only be completely eliminated when the viewing angle and the incident light direction are in particular configuration called the Brewster angle. By using images obtained through a linear polarizer at two or more orientations and applying ICA to them, we can completely separate reflections [15]. Let's take an example of a photograph of a painting behind glassy window. The final image is a linear combination of the light reflected by the painting and the light directly reflected by the glass in front of the painting (Fig. 7). The amount of light at a single point in the image can be expressed as where is the amount of light contributed by the painintg and by the reflection. and are the multiplicative constants. By changing the orientation of the polarizer, we obtain two new multiplicative constants and, because the amount of light from each of the sources changes. That means we have a linear mixing problem where observations are [ ], independent components [ ] and the mixing matrix ( ). Then, we apply ICA to (15) in order to obtain independent components and. The results are shown in Fig. 8. (14) (15) Figure 7. [LEFT] Renoir's On the Terrace, Sheila and Sheila's reflection. [RIGHT] A photograph of a painting behind glass contains a superposition of the light that is reflected by the painting, and the light that is reflected directly off the glass [15]. 11

12 Figure 8. [LEFT] A pair of images of Renoir's On the Terrace with a reflection of Sheila photographed through a linear polarizer at orthogonal orientations. [RIGHT] A pair of ICs [15]. 4 Conclusions We have seen that ICA is a very general purpose and applicative computational method with an almost unlimited potential for future use. Unfortunately, for now, the ICA method is restricted to linear and barely nonlinear problems. However, the research on the field of independent component analysis is still very active nowadays, which gives hope for further improvements of the method and its algorithmic implementations. References [1] J. Herault and C. Jutten, Signal Processing 24, 1-10 (1986). [2] P. Comon, Signal Processing 36, (1994). [3] A.J. Bell and T.J. Sejnowski, Neural Computation 7, (1995). [4] A. Hyvärinen and E. Oja, Neural Networks 13, (2000) [5] A. Hyvärinen, IEEE Trans. Neural Networks 10, (1999). [6] A. Hyvärinen, Neural Computing Surveys 2, (1999). [7] J.B. Fritz, M. Elhilali, Curr. Opin. Neurbiol 17, (2007). [8] A. Papoulis, Probability, Random Variables and Stochastic Processes, 3rd edn. (McGraW-Hill, New York, 1991). [9] S. Širca, A. Horvat, Computational Methods for Physicists (Springer-Verlag, 2012). [10] A.Ciaramella, Nonlinear Processes in Geophysics 11, (2004). [11] F. Aires, A. Chedin and J.P. Nadal, J. Geophys. Res. 105 (D13), (2000). [12] T. Ristaniemi, J. Joutsensalo, Signal Processing 82, (2002). [13] S. Bounlong, D. Lowe, D. Saad, Journal of Machine Learning 4, (2003). [14] T. He, G. Clifford and L. Tarassenko. Neural Comput. & Applic 15, (2006). [15] H. Farid and E.H. Adelson. IEEE Comp. Soc. Conf. on Comp. Vis. and Patt. Rec. 1, (1999). 12