Undercomplete Independent Component. Analysis for Signal Separation and. Dimension Reduction. Category: Algorithms and Architectures.
|
|
- Patrick Gallagher
- 5 years ago
- Views:
Transcription
1 Undercomplete Independent Component Analysis for Signal Separation and Dimension Reduction John Porrill and James V Stone Psychology Department, Sheeld University, Sheeld, S10 2UR, England. Tel: Fax: , J.Porrill, J.V.Stone@shef.ac.uk Category: Algorithms and Architectures. Author for correspondence: JV Stone. Abstract We introduce undercomplete independent component analysis (uica), a method for extracting K signals from M K mixtures of N M source signals. The mixtures x = (x 1 ; : : :; x M ) T are formed from a linear combination of the independent source signals s = (s 1 ; : : :; s N ) T using an M N mixing matrix A, so that x = As. For the case N = K = M, Bell and Sejnowski [Bell and Sejnowski, 1995] showed that a square N N unmixing matrix W can be found by maximising the joint entropy of M signals (Y 1 ; : : :; Y N ) T = (W x), where is a monotonic, non-linear function. Using a similar approach, we show that a K M unmixing matrix W can be used to recover K M signals (s 1 ; : : :; s K ) T by maximising the joint entropy of K signals Y = (W x). The matrix W is essentially a pseudo-inverse of the M N mixing matrix A. A dierent and widely used method for reducing the size of W is to perform principal component analysis (PCA) on the data set, and to use only the L principal components with the largest eigenvalues as input to ICA. This results in an L L unmixing matrix W. However, there is no a priori reason to assume that independent components exist in only the L-D subspace dened by the L principal components with largest eigenvectors. Thus, discarding some eigenvectors may also corrupt or discard independent components. In contrast, uica does not discard any independent components in the data set, and can extract between 1 and M signals from x. The method is demonstrated on mixtures of high kurtosis (speech and music) and Gaussian signals.
2 Introduction We present a method for extracting K signals from M mixtures of N sources, where K M N. This is a generalisation of the method described by Bell and Sejnowski (B&S) [Bell and Sejnowski, 1995], who showed how K signals could be extracted from M mixtures of N source signals for K = M = N. Both methods can be described informally as follows. The amplitudes of N source signals can be represented as a point in an N-dimensional space, and, considered over all times, they dene a distribution of points in this space. If the signals are from dierent sources then they tend to be statistically independent of each other. A key observation is that, if a signal s has a cumulative density function (cdf) then the distribution of (s) has maximum entropy. Similarly, if N signals each have cdf then the joint distribution of (s) = ((s 1 ); : : :; (s N )) T has maximum entropy. For a set of signal mixtures x = As, a linear unmixing matrix exists such that s = W x. Given that (s) has maximum entropy, s can be recovered by nding a matrix W that maximises the entropy of Y = (W x), at which point W x = s. Why Extract Fewer Sources Than Mxtures? Given a temporal sequence x of M (P P ) images, ICA can be used to extract M spatially independent components (IC) with an (M M) unmixing matrix W. However, such a matrix may be large. In such cases, we would like to be able to extract fewer sources than mixtures. A common method for reducing the size of the unmixing matrix W is to perform principal component analysis (PCA) on the data matrix x, and then to retain the L < M principal components (PCs) with the largest eigenvalues. One property of ICA is that it is insensitive to the RMS amplitude of the ICs it extracts from the mixtures x. However, for a given data set x, there is no a priori reason to suppose that ICs should reside only within the subspace dened by the L PCs with the largest eigenvalues. Thus, discarding a subspace using PCA removes ICs that exist within it, and partially destroys any IC with a non-zero projection onto that subspace. For example, in analysing fmri data, an IC with very small variance was found to be associated with the form of the 'on-o' experimental protocol used [McKeown et al., 1998]. Additionally, the reduced dimensionality of the input space dened by the L retained PCs may contain more ICs than PCs. That is, if the original M-D input space x contains N ICs then, as the number L of retained PCs is reduced, there is an increasing likelihood that L < N. If L < N then a linear decomposition of the L eigenvectors into K <= N sources s does not exist, and therefore ICA cannot work in this case 1. Signal Separation Using Entropy Maximisation Suppose that the outputs x = (x 1 ; : : :; x M ) T of M measurement devices are a linear mixture of N independent signal sources s = (s 1 ; : : :; s N ) T, x = As, where A is an M N mixing matrix. We wish to nd a K M unmixing matrix W such that the K recovered components y = W x are a subset of the original signals s (i.e. K N). 1 Thanks to Martin McKeown for pointing this out
3 Signal Separation For Equal Numbers of Sources and Source Mixtures In the case K = M = N, B&S showed that the unmixing matrix W can be found by maximising the entropy H(Y) of the joint distribution Y = fy 1 ; : : :; Y N g = f 1 (y 1 ); : : :; N (y N )g, where y i = W x i. The correct i have the same form as the cdfs of the input signals x i. However, in many cases it is sucient to approximate these cdfs by sigmoids 2 Y i = tanh(y i ). The output entropy H(Y) can be shown to be related to the entropy of the input H(x) by H(Y) = H(x) + E [ log jjj ] (1) where E denotes expected value, and jjj is the absolute value of the determinant of the Jacobian We can evaluate @y are Jacobian matrices. Equation (1) yields H(Y) = H(x) + E Y N = i(y 0 i )jw j (2) " NX Substituting Equation (2) in + log jw j: (3) The term H(x) is constant, P and can therefore be ignored in the maximisation of H(Y). The term E [ log i 0 ] can be estimated given n samples from the distribution dened by y: E " NX 1 n nx NX j=1 (y(j) i ) (4) Ignoring H(x), and substituting Equation (4) in (3) yields a new function that diers from H(Y) by a constant (= H(x)) h(w ) = 1 n nx NX j=1 (y(j) i ) + log jw j (5) B&S showed how maximising this function with respect to the matrix W can be used to recover linear mixtures of signals. Signal Separation For Unequal Numbers of Sources and Source Mixtures The assumption K = M = N is very restrictive. If the number of sources is unknown one might like to reduce the dimensionality of the problem by looking for a small subset of the source signals, so that in general K < M. This means that a rectangular unmixing matrix is required. This can be determined using the criterion of maximum output entropy which ensures unmixing of those independent variables with cdfs that best t the functional forms of i (this follows from the maximumlikelihood interpretation of the maximum entropy criterion, see [Amari et al., 1996]). Note that if K < M then the mere independence of the outputs y is no longer sucient to guarantee that we have recovered a subset of the input variables because all linear combinations of disjoint subsets of inputs are independent. For 2 In fact sources s i normalised so that E[s i tanh s i] = 1=2 can be separated using tanh sigmoids if and only if the pairwise conditions i j > 1 are satised, where i = 2E[s 2 i ]E[sech 2 s i].
4 example, given four signal mixtures of four source signals, it is possible to extract two independent signals (y 1 and y 1 ) that are linear combinations of disjoint pairs of source signals, y 1 = w 1 s 1 + w 2 s 2 and y 2 = w 3 s 3 + w 4 s 4 ; where the w's are elements in a 2 2 unmixing matrix W. Note that, because y 1 and y 2 are mixtures, they are approximately Gaussian. The criterion that the recovered variables have cdf's approximated by i is thus of much greater importance in the case K < M than if K = M. Thus, to simultaneously optimise both W and the i would be counter-productive for K < M. Equation (1) cannot be used when K < M, so we replace it by H(Y) = H(y) + log j@y=@yj: H(Y) = H(y) + E " KX In general, the entropy H(y) is dicult to calculate. We can approximate it by the entropy of a multi-dimensional Gaussian, which is given by H(y) 1=2 log jcj + K=2 (1 + log 2), where C = Cov[y]. However, as the algorithm converges, this approximation becomes less accurate. This is because the projection of the input data onto the subspace dened by the rows of W denes an increasingly non-gaussian distribution as the algorithm converges. In practice, this approximation seems to work adequately (see Results). More accurate approximations involving higher moments of the distribution than the covariance can be derived if required (for example [Amari et al., 1996]). C can be re-written as C = E[yy T ] = W SW T where S = Cov[x]. In maximising H(Y), we can ignore the constant K=2 (1 + log 2). We can now dene a new function which is an approximation to H(Y), and which diers from it by this constant: h(w ) = 1 2 log jw SW T j + E " KX This can be maximised using its derivative, which can be shown to be: ij = W T ij + E [ 00 i =0 i x j] (8) where W = (SW T )(W SW T )?1, which is the pseudo-inverse of W with respect to the positive denite matrix S. Note that, if K = N then W = W?1. If i = tanh then this evaluates to: r W h = W T? 2 E y T x (9) In our experiments, this gradient was used to maximise equation (7), using a BFGS quasi-newton method. Results Extracting K Source Signals from M K Mixtures Using uica: The method has been tested by extracting signals from mixtures of natural sounds and Gaussian noise. The program was stopped when the correlation r between each of the K outputs and exactly one of the N input signals s was greater than 0:95, or when the number of iterations (evaluations of h(y)) exceeded All signals were normalised to have zero mean and unit variance. The N signals were then linearly combined using a random M N mixing matrix (with normally distributed independent entries) to produce M signal mixtures x, which were used as input to the method. In experiments reported here N = 6. The signals (s 1 ; s 2 ; s 3 ) were a gong, Handel's Messiah, and laughter, respectively, obtained from the MatLab
5 software package. Each of the signals s 4 to s 6 was a dierent random sample of Gaussian noise. Each source signal consisted of a random sample of 20,000 points from each signal. The rst task consists of extracting K = 1; 3 and 4 dierent sound signals from M = 6 linear mixtures of N = 6 signals. For K = 3 all and only the three non- Gaussian sources were recovered, despite dierent initial values for W. For K = 4, these three sources were always recovered, with the highest correlation between one output and a Gaussian source being around 0:8. For K = 1, the algorithm was run three times with dierent random number seeds. On each occasion, s 1 was recovered; this source having the distribution with most kurtosis, so that its cdf is a good match to the tanh non-linearity. The results are displayed in Table 1. These experiments were repeated with M = 12 and 24, with K = 3 and 1. In each case, the required number of sources was recovered, each source was recovered once only, and these sources did not include one of the Gaussian sources. Extracting K Source Signals from M K Mixtures Using PCA/ICA: We compared the results obtained with uica with those obtained using a conventional dimension reduction method (PCA) to preprocess the set of M mixtures. The results presented here involve the three sound sources described in the previous section, plus three other sound sources (obtained from MatLab). Each source consisted of 10,000 samples due to the short length of one source signal. All signals were normalised and mixed together as described in the previous section. First, we ran uica with six mixtures, and set the number of required sources to K = 4. The four extracted signals each had a correlation of jrj > 0:9 with exactly one of the source signals (see Table 2). This result is consistent with those reported in the previous section. Next, we ran ICA after preprocessing with PCA to obtain four eigenvectors with the largest eigenvalues. These were then used as input to ICA using a 4 4 unmixing matrix. From Table 3, only three sources (1,3 and 6) can reasonably considered to have been extracted, with jrj > 0:9. The remaining sources (2, 4 and 5) have maximum correlations with extracted signals of 0.56, 0.59 and 0.50, respectively. Thus, one IC had been eectively discarded along with the two eigenvectors with smallest eigenvalues. For completeness, all six eigenvectors and a 6 6 unmixing matrix were used. In this case, each source signal had a correlation jrj > 0:95 with exactly one extracted signal and jrj < 0:25 with the remaining extracted signals. These results demonstrate that using PCA to reduce the number of signal mixtures used as input to ICA can compromise the ability of ICA to extract source signals. This is because the source signals had non-zero projections on to the discarded eigenvectors. In contrast, uica does not require PCA, and can therefore extract exactly K N source signals from M N signal mixtures. Discussion Two alternative approaches to the problem of dimension reduction are: 1) reduce the dimension of the mixture space from M to K before separating signals, which risks corrupting ICs, or, 2) nd K = M > N components (which could be a large number) using the B&S's method, and then apply a separate method to identify the K < M most important components (see [Cichocki and Kasprzak, 1996]). With respect to 1, discarding PCs cannot also inadvertently discard ICs if only PCs with
6 zero eigenvalues are discarded. However, zero eigenvalues are rarely encountered with noisy data, so that one is forced to risk corrupting ICs when using PCA to reduce the dimensionality of data used as input to ICA. The method we have described unies approaches 1 and 2, without compromising the ability of ICA to extract ICs. A logical modication to our algorithm would be to extract ICs one at a time, as in [Girolami and Fyfe, 1996] using projection pursuit indices. This involves extracting one IC from the M-dimensional space of signal mixtures x, 'removing' the one-dimensional subspace corresponding to that IC (using Gramm-Schmidt orthonormalisation), and then extracting the next IC. This operation is repeated until all the sources have been extracted. This method has been tested on the data described above, and results are not noticably dierent from those reported here. However, we conjecture that sequential extraction of sources does not provide similar results to ICA in general. Consider image data which is a mixture of two spatial ICs each of which has exactly one region (A and B, respectively) with non-zero grey-levels, and the same grey-levels in an overlapping region C of the IC images. The independence criterion implict in ICA would force it to identify three ICs, corresponding to regions (A? C), (B? C) and C. In contrast, the `cdf-matching' criterion implicit in projection pursuit methods (and uica for K = 1) would ensure that an IC corresponding to A would be extracted rst, followed by B 3. Conclusion Using an undercomplete basis set to extract ICs is useful when the number of signal mixtures is larger than the number of source signals. This commonly occurs in high dimensional data sets in which each signal is an entire image, or even a sequence of images. The obvious strategy of discarding PCs of the data set that have small eigenvalues can compromise ICA's ability to extract ICs for two reasons. First, ICs may have non-zero projections onto the subspace dened by these discarded PCs; so partially destroying these ICs. Second, ICA is only possible if the number of source signals is equal to or less than the number of signal mixtures. Using PCA to eectively reduce the number of mixtures therefore increases the probability that the number of source signals is greater than the number of mixtures. In contrast, uica extracts a specied number of sources signals from the original data set, and therefore precludes the problems associated with preprocessing with PCA. Acknowledgements: Thanks to members of the Computational Neurobiology Laboratory at the Salk Institute, and to Tony Bell, for comments on this work. Thanks to Stephen Isard for comments on a previous draft of this paper. J Stone is supported by a Mathematical Biology Wellcome Fellowship (Grant number ). References [Amari et al., 1996] Amari, S., Cichocki, A., and Yang, H. (1996). A new learning algorithm for blind signal separation. In Touretzky, D., Mozer, M., and Hasselmo, M., editors, Advances in Neural Information Processing Systems 8. MIT Press, Cambridge MA (In Press). [Bell and Sejnowski, 1995] Bell, A. and Sejnowski, T. (1995). An informationmaximization approach to blind separation and blind deconvolution. Neural Computation, 7:1129{ Thanks for Martin McKeown for pointing this out
7 [Cichocki and Kasprzak, 1996] Cichocki, A. and Kasprzak, W. (1996). Local adapative learning algorithms for blind separation of natural images. Neural Network World, 6(4):515{523. [Girolami and Fyfe, 1996] Girolami, M. and Fyfe, C. (1996). Negentropy and kurtosis as projection pursuit indices provide generalised ica algorithms. NIPS96 Blind Signal Separation Workshop. [McKeown et al., 1998] McKeown, M., Makeig, S., Brown, G., Jung, T., Kindermann, S., and Sejnowski, T. (1998). Spatially independent activity patterns in functional magnetic resonance imaging data during the stroop color-naming task. Proceedings of the National Academy of Sciences USA., 95:803{810. Sources (N) Mixtures (M) Required (K) Extracted Iterations Table 1: Performance for N=6 signals s = (s 1 ; : : :; s 6 ). (s 1 ; s 2 ; s 3 ) are a gong, Handel's Messiah, and laughter, respectively, and (s 4 ; s 5 ; s 6 ) are three Gaussian noise signals. The method was tested with dierent numbers M of signal mixtures and dierent numbers K of required outputs. Iterations denotes the number of function evaluations of h(w ) required for convergence (see text). Src 1 Src 2 Src 3 Src 4 Src 5 Src Table 2: Using uica to extract 4 signals from a mixture of 6 sound signals. Each cell species the absolute value of the correlation jrj between a source signal (columns) and a signal extracted by uica (rows). Src 1 Src 2 Src 3 Src 4 Src 5 Src Table 3: Using PCA to preprocess six mixtures to obtain four PCs. Each cell species the absolute value of the correlation jrj between a source signal (columns) and an extracted signal (rows). The four PCs were used as input to ICA. Only three of the source signals (in bold typeface) can be considered to have been extracted, with correlations jrj > 0:90.
File: ica tutorial2.tex. James V Stone and John Porrill, Psychology Department, Sheeld University, Tel: Fax:
File: ica tutorial2.tex Independent Component Analysis and Projection Pursuit: A Tutorial Introduction James V Stone and John Porrill, Psychology Department, Sheeld University, Sheeld, S 2UR, England.
More informationIndependent Component Analysis
Independent Component Analysis James V. Stone November 4, 24 Sheffield University, Sheffield, UK Keywords: independent component analysis, independence, blind source separation, projection pursuit, complexity
More informationICA [6] ICA) [7, 8] ICA ICA ICA [9, 10] J-F. Cardoso. [13] Matlab ICA. Comon[3], Amari & Cardoso[4] ICA ICA
16 1 (Independent Component Analysis: ICA) 198 9 ICA ICA ICA 1 ICA 198 Jutten Herault Comon[3], Amari & Cardoso[4] ICA Comon (PCA) projection persuit projection persuit ICA ICA ICA 1 [1] [] ICA ICA EEG
More informationNatural Gradient Learning for Over- and Under-Complete Bases in ICA
NOTE Communicated by Jean-François Cardoso Natural Gradient Learning for Over- and Under-Complete Bases in ICA Shun-ichi Amari RIKEN Brain Science Institute, Wako-shi, Hirosawa, Saitama 351-01, Japan Independent
More informationIndependent Component Analysis
A Short Introduction to Independent Component Analysis Aapo Hyvärinen Helsinki Institute for Information Technology and Depts of Computer Science and Psychology University of Helsinki Problem of blind
More information1 Introduction Independent component analysis (ICA) [10] is a statistical technique whose main applications are blind source separation, blind deconvo
The Fixed-Point Algorithm and Maximum Likelihood Estimation for Independent Component Analysis Aapo Hyvarinen Helsinki University of Technology Laboratory of Computer and Information Science P.O.Box 5400,
More informationIndependent component analysis: an introduction
Research Update 59 Techniques & Applications Independent component analysis: an introduction James V. Stone Independent component analysis (ICA) is a method for automatically identifying the underlying
More informationA Constrained EM Algorithm for Independent Component Analysis
LETTER Communicated by Hagai Attias A Constrained EM Algorithm for Independent Component Analysis Max Welling Markus Weber California Institute of Technology, Pasadena, CA 91125, U.S.A. We introduce a
More informationFundamentals of Principal Component Analysis (PCA), Independent Component Analysis (ICA), and Independent Vector Analysis (IVA)
Fundamentals of Principal Component Analysis (PCA),, and Independent Vector Analysis (IVA) Dr Mohsen Naqvi Lecturer in Signal and Information Processing, School of Electrical and Electronic Engineering,
More informationHST.582J/6.555J/16.456J
Blind Source Separation: PCA & ICA HST.582J/6.555J/16.456J Gari D. Clifford gari [at] mit. edu http://www.mit.edu/~gari G. D. Clifford 2005-2009 What is BSS? Assume an observation (signal) is a linear
More informationIndependent Component Analysis of Incomplete Data
Independent Component Analysis of Incomplete Data Max Welling Markus Weber California Institute of Technology 136-93 Pasadena, CA 91125 fwelling,rmwg@vision.caltech.edu Keywords: EM, Missing Data, ICA
More informationCIFAR Lectures: Non-Gaussian statistics and natural images
CIFAR Lectures: Non-Gaussian statistics and natural images Dept of Computer Science University of Helsinki, Finland Outline Part I: Theory of ICA Definition and difference to PCA Importance of non-gaussianity
More informationbelow, kernel PCA Eigenvectors, and linear combinations thereof. For the cases where the pre-image does exist, we can provide a means of constructing
Kernel PCA Pattern Reconstruction via Approximate Pre-Images Bernhard Scholkopf, Sebastian Mika, Alex Smola, Gunnar Ratsch, & Klaus-Robert Muller GMD FIRST, Rudower Chaussee 5, 12489 Berlin, Germany fbs,
More informationIndependent Component Analysis
1 Independent Component Analysis Background paper: http://www-stat.stanford.edu/ hastie/papers/ica.pdf 2 ICA Problem X = AS where X is a random p-vector representing multivariate input measurements. S
More informationPrincipal Component Analysis
Principal Component Analysis Introduction Consider a zero mean random vector R n with autocorrelation matri R = E( T ). R has eigenvectors q(1),,q(n) and associated eigenvalues λ(1) λ(n). Let Q = [ q(1)
More informationGatsby Theoretical Neuroscience Lectures: Non-Gaussian statistics and natural images Parts I-II
Gatsby Theoretical Neuroscience Lectures: Non-Gaussian statistics and natural images Parts I-II Gatsby Unit University College London 27 Feb 2017 Outline Part I: Theory of ICA Definition and difference
More informationIndependent Component Analysis (ICA) Bhaskar D Rao University of California, San Diego
Independent Component Analysis (ICA) Bhaskar D Rao University of California, San Diego Email: brao@ucsdedu References 1 Hyvarinen, A, Karhunen, J, & Oja, E (2004) Independent component analysis (Vol 46)
More informationIndependent Component Analysis
A Short Introduction to Independent Component Analysis with Some Recent Advances Aapo Hyvärinen Dept of Computer Science Dept of Mathematics and Statistics University of Helsinki Problem of blind source
More informationIndependent Component Analysis and Its Applications. By Qing Xue, 10/15/2004
Independent Component Analysis and Its Applications By Qing Xue, 10/15/2004 Outline Motivation of ICA Applications of ICA Principles of ICA estimation Algorithms for ICA Extensions of basic ICA framework
More informationPCA & ICA. CE-717: Machine Learning Sharif University of Technology Spring Soleymani
PCA & ICA CE-717: Machine Learning Sharif University of Technology Spring 2015 Soleymani Dimensionality Reduction: Feature Selection vs. Feature Extraction Feature selection Select a subset of a given
More informationAdvanced Introduction to Machine Learning CMU-10715
Advanced Introduction to Machine Learning CMU-10715 Independent Component Analysis Barnabás Póczos Independent Component Analysis 2 Independent Component Analysis Model original signals Observations (Mixtures)
More informationPROPERTIES OF THE EMPIRICAL CHARACTERISTIC FUNCTION AND ITS APPLICATION TO TESTING FOR INDEPENDENCE. Noboru Murata
' / PROPERTIES OF THE EMPIRICAL CHARACTERISTIC FUNCTION AND ITS APPLICATION TO TESTING FOR INDEPENDENCE Noboru Murata Waseda University Department of Electrical Electronics and Computer Engineering 3--
More informationHST.582J / 6.555J / J Biomedical Signal and Image Processing Spring 2007
MIT OpenCourseWare http://ocw.mit.edu HST.582J / 6.555J / 16.456J Biomedical Signal and Image Processing Spring 2007 For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.
More informationGaussian process for nonstationary time series prediction
Computational Statistics & Data Analysis 47 (2004) 705 712 www.elsevier.com/locate/csda Gaussian process for nonstationary time series prediction Soane Brahim-Belhouari, Amine Bermak EEE Department, Hong
More informationIndependent Component Analysis. Contents
Contents Preface xvii 1 Introduction 1 1.1 Linear representation of multivariate data 1 1.1.1 The general statistical setting 1 1.1.2 Dimension reduction methods 2 1.1.3 Independence as a guiding principle
More informationLinear Regression and Its Applications
Linear Regression and Its Applications Predrag Radivojac October 13, 2014 Given a data set D = {(x i, y i )} n the objective is to learn the relationship between features and the target. We usually start
More information1 Introduction Blind source separation (BSS) is a fundamental problem which is encountered in a variety of signal processing problems where multiple s
Blind Separation of Nonstationary Sources in Noisy Mixtures Seungjin CHOI x1 and Andrzej CICHOCKI y x Department of Electrical Engineering Chungbuk National University 48 Kaeshin-dong, Cheongju Chungbuk
More informationCheng Soon Ong & Christian Walder. Canberra February June 2018
Cheng Soon Ong & Christian Walder Research Group and College of Engineering and Computer Science Canberra February June 2018 Outlines Overview Introduction Linear Algebra Probability Linear Regression
More informationAn Improved Cumulant Based Method for Independent Component Analysis
An Improved Cumulant Based Method for Independent Component Analysis Tobias Blaschke and Laurenz Wiskott Institute for Theoretical Biology Humboldt University Berlin Invalidenstraße 43 D - 0 5 Berlin Germany
More informationNon-Euclidean Independent Component Analysis and Oja's Learning
Non-Euclidean Independent Component Analysis and Oja's Learning M. Lange 1, M. Biehl 2, and T. Villmann 1 1- University of Appl. Sciences Mittweida - Dept. of Mathematics Mittweida, Saxonia - Germany 2-
More informationICA. Independent Component Analysis. Zakariás Mátyás
ICA Independent Component Analysis Zakariás Mátyás Contents Definitions Introduction History Algorithms Code Uses of ICA Definitions ICA Miture Separation Signals typical signals Multivariate statistics
More informationIndependent Component Analysis (ICA)
Independent Component Analysis (ICA) Université catholique de Louvain (Belgium) Machine Learning Group http://www.dice.ucl ucl.ac.be/.ac.be/mlg/ 1 Overview Uncorrelation vs Independence Blind source separation
More informationStatistical Analysis of fmrl Data
Statistical Analysis of fmrl Data F. Gregory Ashby The MIT Press Cambridge, Massachusetts London, England Preface xi Acronyms xv 1 Introduction 1 What Is fmri? 2 The Scanning Session 4 Experimental Design
More informationIndependent Component Analysis Using an Extended Infomax Algorithm for Mixed Subgaussian and Supergaussian Sources
LETTER Communicated by Jean-François Cardoso Independent Component Analysis Using an Extended Infomax Algorithm for Mixed Subgaussian and Supergaussian Sources Te-Won Lee Howard Hughes Medical Institute,
More informationArtificial Intelligence Module 2. Feature Selection. Andrea Torsello
Artificial Intelligence Module 2 Feature Selection Andrea Torsello We have seen that high dimensional data is hard to classify (curse of dimensionality) Often however, the data does not fill all the space
More informationIntroduction to Independent Component Analysis. Jingmei Lu and Xixi Lu. Abstract
Final Project 2//25 Introduction to Independent Component Analysis Abstract Independent Component Analysis (ICA) can be used to solve blind signal separation problem. In this article, we introduce definition
More informationMassoud BABAIE-ZADEH. Blind Source Separation (BSS) and Independent Componen Analysis (ICA) p.1/39
Blind Source Separation (BSS) and Independent Componen Analysis (ICA) Massoud BABAIE-ZADEH Blind Source Separation (BSS) and Independent Componen Analysis (ICA) p.1/39 Outline Part I Part II Introduction
More informationTo appear in Proceedings of the ICA'99, Aussois, France, A 2 R mn is an unknown mixture matrix of full rank, v(t) is the vector of noises. The
To appear in Proceedings of the ICA'99, Aussois, France, 1999 1 NATURAL GRADIENT APPROACH TO BLIND SEPARATION OF OVER- AND UNDER-COMPLETE MIXTURES L.-Q. Zhang, S. Amari and A. Cichocki Brain-style Information
More informationADAPTIVE LATERAL INHIBITION FOR NON-NEGATIVE ICA. Mark Plumbley
Submitteed to the International Conference on Independent Component Analysis and Blind Signal Separation (ICA2) ADAPTIVE LATERAL INHIBITION FOR NON-NEGATIVE ICA Mark Plumbley Audio & Music Lab Department
More informationLearning with Ensembles: How. over-tting can be useful. Anders Krogh Copenhagen, Denmark. Abstract
Published in: Advances in Neural Information Processing Systems 8, D S Touretzky, M C Mozer, and M E Hasselmo (eds.), MIT Press, Cambridge, MA, pages 190-196, 1996. Learning with Ensembles: How over-tting
More informationSeparation of the EEG Signal using Improved FastICA Based on Kurtosis Contrast Function
Australian Journal of Basic and Applied Sciences, 5(9): 2152-2156, 211 ISSN 1991-8178 Separation of the EEG Signal using Improved FastICA Based on Kurtosis Contrast Function 1 Tahir Ahmad, 2 Hjh.Norma
More informationSingle Channel Signal Separation Using MAP-based Subspace Decomposition
Single Channel Signal Separation Using MAP-based Subspace Decomposition Gil-Jin Jang, Te-Won Lee, and Yung-Hwan Oh 1 Spoken Language Laboratory, Department of Computer Science, KAIST 373-1 Gusong-dong,
More informationStatistical Machine Learning
Statistical Machine Learning Christoph Lampert Spring Semester 2015/2016 // Lecture 12 1 / 36 Unsupervised Learning Dimensionality Reduction 2 / 36 Dimensionality Reduction Given: data X = {x 1,..., x
More informationICA and ISA Using Schweizer-Wolff Measure of Dependence
Keywords: independent component analysis, independent subspace analysis, copula, non-parametric estimation of dependence Abstract We propose a new algorithm for independent component and independent subspace
More informationPlan of Class 4. Radial Basis Functions with moving centers. Projection Pursuit Regression and ridge. Principal Component Analysis: basic ideas
Plan of Class 4 Radial Basis Functions with moving centers Multilayer Perceptrons Projection Pursuit Regression and ridge functions approximation Principal Component Analysis: basic ideas Radial Basis
More informationIndependent Component Analysis
Independent Component Analysis Seungjin Choi Department of Computer Science Pohang University of Science and Technology, Korea seungjin@postech.ac.kr March 4, 2009 1 / 78 Outline Theory and Preliminaries
More informationUnsupervised learning: beyond simple clustering and PCA
Unsupervised learning: beyond simple clustering and PCA Liza Rebrova Self organizing maps (SOM) Goal: approximate data points in R p by a low-dimensional manifold Unlike PCA, the manifold does not have
More informationIndependent Components Analysis
CS229 Lecture notes Andrew Ng Part XII Independent Components Analysis Our next topic is Independent Components Analysis (ICA). Similar to PCA, this will find a new basis in which to represent our data.
More informationBlind Machine Separation Te-Won Lee
Blind Machine Separation Te-Won Lee University of California, San Diego Institute for Neural Computation Blind Machine Separation Problem we want to solve: Single microphone blind source separation & deconvolution
More informationRigid Structure from Motion from a Blind Source Separation Perspective
Noname manuscript No. (will be inserted by the editor) Rigid Structure from Motion from a Blind Source Separation Perspective Jeff Fortuna Aleix M. Martinez Received: date / Accepted: date Abstract We
More informationSoft-LOST: EM on a Mixture of Oriented Lines
Soft-LOST: EM on a Mixture of Oriented Lines Paul D. O Grady and Barak A. Pearlmutter Hamilton Institute National University of Ireland Maynooth Co. Kildare Ireland paul.ogrady@may.ie barak@cs.may.ie Abstract.
More informationGaussian Processes for Regression. Carl Edward Rasmussen. Department of Computer Science. Toronto, ONT, M5S 1A4, Canada.
In Advances in Neural Information Processing Systems 8 eds. D. S. Touretzky, M. C. Mozer, M. E. Hasselmo, MIT Press, 1996. Gaussian Processes for Regression Christopher K. I. Williams Neural Computing
More informationAn Introduction to Independent Components Analysis (ICA)
An Introduction to Independent Components Analysis (ICA) Anish R. Shah, CFA Northfield Information Services Anish@northinfo.com Newport Jun 6, 2008 1 Overview of Talk Review principal components Introduce
More informationOn Information Maximization and Blind Signal Deconvolution
On Information Maximization and Blind Signal Deconvolution A Röbel Technical University of Berlin, Institute of Communication Sciences email: roebel@kgwtu-berlinde Abstract: In the following paper we investigate
More informationRemaining energy on log scale Number of linear PCA components
NONLINEAR INDEPENDENT COMPONENT ANALYSIS USING ENSEMBLE LEARNING: EXPERIMENTS AND DISCUSSION Harri Lappalainen, Xavier Giannakopoulos, Antti Honkela, and Juha Karhunen Helsinki University of Technology,
More informationRobust extraction of specific signals with temporal structure
Robust extraction of specific signals with temporal structure Zhi-Lin Zhang, Zhang Yi Computational Intelligence Laboratory, School of Computer Science and Engineering, University of Electronic Science
More informationc Springer, Reprinted with permission.
Zhijian Yuan and Erkki Oja. A FastICA Algorithm for Non-negative Independent Component Analysis. In Puntonet, Carlos G.; Prieto, Alberto (Eds.), Proceedings of the Fifth International Symposium on Independent
More information2 Tikhonov Regularization and ERM
Introduction Here we discusses how a class of regularization methods originally designed to solve ill-posed inverse problems give rise to regularized learning algorithms. These algorithms are kernel methods
More informationRecursive Generalized Eigendecomposition for Independent Component Analysis
Recursive Generalized Eigendecomposition for Independent Component Analysis Umut Ozertem 1, Deniz Erdogmus 1,, ian Lan 1 CSEE Department, OGI, Oregon Health & Science University, Portland, OR, USA. {ozertemu,deniz}@csee.ogi.edu
More informationChapter 15 - BLIND SOURCE SEPARATION:
HST-582J/6.555J/16.456J Biomedical Signal and Image Processing Spr ing 2005 Chapter 15 - BLIND SOURCE SEPARATION: Principal & Independent Component Analysis c G.D. Clifford 2005 Introduction In this chapter
More informationMulti-Class Linear Dimension Reduction by. Weighted Pairwise Fisher Criteria
Multi-Class Linear Dimension Reduction by Weighted Pairwise Fisher Criteria M. Loog 1,R.P.W.Duin 2,andR.Haeb-Umbach 3 1 Image Sciences Institute University Medical Center Utrecht P.O. Box 85500 3508 GA
More informationCONVOLUTIVE NON-NEGATIVE MATRIX FACTORISATION WITH SPARSENESS CONSTRAINT
CONOLUTIE NON-NEGATIE MATRIX FACTORISATION WITH SPARSENESS CONSTRAINT Paul D. O Grady Barak A. Pearlmutter Hamilton Institute National University of Ireland, Maynooth Co. Kildare, Ireland. ABSTRACT Discovering
More informationA Canonical Genetic Algorithm for Blind Inversion of Linear Channels
A Canonical Genetic Algorithm for Blind Inversion of Linear Channels Fernando Rojas, Jordi Solé-Casals, Enric Monte-Moreno 3, Carlos G. Puntonet and Alberto Prieto Computer Architecture and Technology
More informationExtraction of Sleep-Spindles from the Electroencephalogram (EEG)
Extraction of Sleep-Spindles from the Electroencephalogram (EEG) Allan Kardec Barros Bio-mimetic Control Research Center, RIKEN, 2271-13 Anagahora, Shimoshidami, Moriyama-ku, Nagoya 463, Japan Roman Rosipal,
More informationDifferent Estimation Methods for the Basic Independent Component Analysis Model
Washington University in St. Louis Washington University Open Scholarship Arts & Sciences Electronic Theses and Dissertations Arts & Sciences Winter 12-2018 Different Estimation Methods for the Basic Independent
More informationCS281 Section 4: Factor Analysis and PCA
CS81 Section 4: Factor Analysis and PCA Scott Linderman At this point we have seen a variety of machine learning models, with a particular emphasis on models for supervised learning. In particular, we
More informationLecture 10: Dimension Reduction Techniques
Lecture 10: Dimension Reduction Techniques Radu Balan Department of Mathematics, AMSC, CSCAMM and NWC University of Maryland, College Park, MD April 17, 2018 Input Data It is assumed that there is a set
More informationwhere A 2 IR m n is the mixing matrix, s(t) is the n-dimensional source vector (n» m), and v(t) is additive white noise that is statistically independ
BLIND SEPARATION OF NONSTATIONARY AND TEMPORALLY CORRELATED SOURCES FROM NOISY MIXTURES Seungjin CHOI x and Andrzej CICHOCKI y x Department of Electrical Engineering Chungbuk National University, KOREA
More informationMatching the dimensionality of maps with that of the data
Matching the dimensionality of maps with that of the data COLIN FYFE Applied Computational Intelligence Research Unit, The University of Paisley, Paisley, PA 2BE SCOTLAND. Abstract Topographic maps are
More informationTutorial on Blind Source Separation and Independent Component Analysis
Tutorial on Blind Source Separation and Independent Component Analysis Lucas Parra Adaptive Image & Signal Processing Group Sarnoff Corporation February 09, 2002 Linear Mixtures... problem statement...
More informationUndercomplete Blind Subspace Deconvolution via Linear Prediction
Undercomplete Blind Subspace Deconvolution via Linear Prediction Zoltán Szabó, Barnabás Póczos, and András L rincz Department of Information Systems, Eötvös Loránd University, Pázmány P. sétány 1/C, Budapest
More information(a)
Chapter 8 Subspace Methods 8. Introduction Principal Component Analysis (PCA) is applied to the analysis of time series data. In this context we discuss measures of complexity and subspace methods for
More informationLECTURE :ICA. Rita Osadchy. Based on Lecture Notes by A. Ng
LECURE :ICA Rita Osadchy Based on Lecture Notes by A. Ng Cocktail Party Person 1 2 s 1 Mike 2 s 3 Person 3 1 Mike 1 s 2 Person 2 3 Mike 3 microphone signals are mied speech signals 1 2 3 ( t) ( t) ( t)
More informationIndependent component analysis: algorithms and applications
PERGAMON Neural Networks 13 (2000) 411 430 Invited article Independent component analysis: algorithms and applications A. Hyvärinen, E. Oja* Neural Networks Research Centre, Helsinki University of Technology,
More informationFinal Report For Undergraduate Research Opportunities Project Name: Biomedical Signal Processing in EEG. Zhang Chuoyao 1 and Xu Jianxin 2
ABSTRACT Final Report For Undergraduate Research Opportunities Project Name: Biomedical Signal Processing in EEG Zhang Chuoyao 1 and Xu Jianxin 2 Department of Electrical and Computer Engineering, National
More informationIndependent Component Analysis and Its Application on Accelerator Physics
Independent Component Analysis and Its Application on Accelerator Physics Xiaoying Pang LA-UR-12-20069 ICA and PCA Similarities: Blind source separation method (BSS) no model Observed signals are linear
More informationCOMS 4721: Machine Learning for Data Science Lecture 19, 4/6/2017
COMS 4721: Machine Learning for Data Science Lecture 19, 4/6/2017 Prof. John Paisley Department of Electrical Engineering & Data Science Institute Columbia University PRINCIPAL COMPONENT ANALYSIS DIMENSIONALITY
More informationOutline Introduction: Problem Description Diculties Algebraic Structure: Algebraic Varieties Rank Decient Toeplitz Matrices Constructing Lower Rank St
Structured Lower Rank Approximation by Moody T. Chu (NCSU) joint with Robert E. Funderlic (NCSU) and Robert J. Plemmons (Wake Forest) March 5, 1998 Outline Introduction: Problem Description Diculties Algebraic
More informationComparative Performance Analysis of Three Algorithms for Principal Component Analysis
84 R. LANDQVIST, A. MOHAMMED, COMPARATIVE PERFORMANCE ANALYSIS OF THR ALGORITHMS Comparative Performance Analysis of Three Algorithms for Principal Component Analysis Ronnie LANDQVIST, Abbas MOHAMMED Dept.
More informationOne-unit Learning Rules for Independent Component Analysis
One-unit Learning Rules for Independent Component Analysis Aapo Hyvarinen and Erkki Oja Helsinki University of Technology Laboratory of Computer and Information Science Rakentajanaukio 2 C, FIN-02150 Espoo,
More informationMULTICHANNEL BLIND SEPARATION AND. Scott C. Douglas 1, Andrzej Cichocki 2, and Shun-ichi Amari 2
MULTICHANNEL BLIND SEPARATION AND DECONVOLUTION OF SOURCES WITH ARBITRARY DISTRIBUTIONS Scott C. Douglas 1, Andrzej Cichoci, and Shun-ichi Amari 1 Department of Electrical Engineering, University of Utah
More informationAn Adaptive Bayesian Network for Low-Level Image Processing
An Adaptive Bayesian Network for Low-Level Image Processing S P Luttrell Defence Research Agency, Malvern, Worcs, WR14 3PS, UK. I. INTRODUCTION Probability calculus, based on the axioms of inference, Cox
More informationOn the INFOMAX algorithm for blind signal separation
University of Wollongong Research Online Faculty of Informatics - Papers (Archive) Faculty of Engineering and Information Sciences 2000 On the INFOMAX algorithm for blind signal separation Jiangtao Xi
More informationBlind Source Separation Using Artificial immune system
American Journal of Engineering Research (AJER) e-issn : 2320-0847 p-issn : 2320-0936 Volume-03, Issue-02, pp-240-247 www.ajer.org Research Paper Open Access Blind Source Separation Using Artificial immune
More informationBlind Extraction of Singularly Mixed Source Signals
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL 11, NO 6, NOVEMBER 2000 1413 Blind Extraction of Singularly Mixed Source Signals Yuanqing Li, Jun Wang, Senior Member, IEEE, and Jacek M Zurada, Fellow, IEEE Abstract
More informationEntropy Manipulation of Arbitrary Non I inear Map pings
Entropy Manipulation of Arbitrary Non I inear Map pings John W. Fisher I11 JosC C. Principe Computational NeuroEngineering Laboratory EB, #33, PO Box 116130 University of Floridaa Gainesville, FL 326 1
More informationFast blind separation based on information theory.
Proceedings 1 YY5 Internatiotlal Symposium un Non-1 i neat- Thec~r-y and Applications 1 :43-47. (I ti press) Fast blind separation based on information theory. Anthony J. Bell & Terrence J. Sejnowski Computational
More informationMark Gales October y (x) x 1. x 2 y (x) Inputs. Outputs. x d. y (x) Second Output layer layer. layer.
University of Cambridge Engineering Part IIB & EIST Part II Paper I0: Advanced Pattern Processing Handouts 4 & 5: Multi-Layer Perceptron: Introduction and Training x y (x) Inputs x 2 y (x) 2 Outputs x
More informationLocalization of Multiple Deep Epileptic Sources in a Realistic Head Model via Independent Component Analysis
Localization of Multiple Deep Epileptic Sources in a Realistic Head Model via Independent Component Analysis David Weinstein, Leonid Zhukov, Geoffrey Potts Email: dmw@cs.utah.edu, zhukov@cs.utah.edu, gpotts@rice.edu
More informationPCA, Kernel PCA, ICA
PCA, Kernel PCA, ICA Learning Representations. Dimensionality Reduction. Maria-Florina Balcan 04/08/2015 Big & High-Dimensional Data High-Dimensions = Lot of Features Document classification Features per
More informationBLIND SEPARATION OF POSITIVE SOURCES USING NON-NEGATIVE PCA
BLIND SEPARATION OF POSITIVE SOURCES USING NON-NEGATIVE PCA Erkki Oja Neural Networks Research Centre Helsinki University of Technology P.O.Box 54, 215 HUT, Finland erkki.oja@hut.fi Mark Plumbley Department
More informationARTEFACT DETECTION IN ASTROPHYSICAL IMAGE DATA USING INDEPENDENT COMPONENT ANALYSIS. Maria Funaro, Erkki Oja, and Harri Valpola
ARTEFACT DETECTION IN ASTROPHYSICAL IMAGE DATA USING INDEPENDENT COMPONENT ANALYSIS Maria Funaro, Erkki Oja, and Harri Valpola Neural Networks Research Centre, Helsinki University of Technology P.O.Box
More informationIntroduction to Machine Learning
10-701 Introduction to Machine Learning PCA Slides based on 18-661 Fall 2018 PCA Raw data can be Complex, High-dimensional To understand a phenomenon we measure various related quantities If we knew what
More informationViewpoint invariant face recognition using independent component analysis and attractor networks
Viewpoint invariant face recognition using independent component analysis and attractor networks Marian Stewart Bartlett University of California San Diego The Salk Institute La Jolla, CA 92037 marni@salk.edu
More informationCh 4. Linear Models for Classification
Ch 4. Linear Models for Classification Pattern Recognition and Machine Learning, C. M. Bishop, 2006. Department of Computer Science and Engineering Pohang University of Science and echnology 77 Cheongam-ro,
More informationBy choosing to view this document, you agree to all provisions of the copyright laws protecting it.
This material is posted here with permission of the IEEE. Such permission of the IEEE does not in any way imply IEEE endorsement of any of Helsinki University of Technology's products or services. Internal
More informationNew Machine Learning Methods for Neuroimaging
New Machine Learning Methods for Neuroimaging Gatsby Computational Neuroscience Unit University College London, UK Dept of Computer Science University of Helsinki, Finland Outline Resting-state networks
More informationCPSC 340: Machine Learning and Data Mining. More PCA Fall 2017
CPSC 340: Machine Learning and Data Mining More PCA Fall 2017 Admin Assignment 4: Due Friday of next week. No class Monday due to holiday. There will be tutorials next week on MAP/PCA (except Monday).
More informationNONLINEAR INDEPENDENT FACTOR ANALYSIS BY HIERARCHICAL MODELS
NONLINEAR INDEPENDENT FACTOR ANALYSIS BY HIERARCHICAL MODELS Harri Valpola, Tomas Östman and Juha Karhunen Helsinki University of Technology, Neural Networks Research Centre P.O. Box 5400, FIN-02015 HUT,
More information