Independent Component Analysis. Contents

Transcription

1 Contents Preface xvii 1 Introduction Linear representation of multivariate data The general statistical setting Dimension reduction methods Independence as a guiding principle Blind source separation Observing mixtures of unknown signals Source separation based on independence Independent component analysis Definition Applications How to find the independent components History of ICA 11 Copyright c 2001 John Wiley & Sons v All rights reserved

2 Part I MATHEMATICAL PRELIMINARIES 2 Random Vectors and Independence Probability distributions and densities Distribution of a random variable Distribution of a random vector Joint and marginal distributions Expectations and moments Definition and general properties Mean vector and correlation matrix Covariances and joint moments Estimation of expectations Uncorrelatedness and independence Uncorrelatedness and whiteness Statistical independence Conditional densities and Bayes rule The multivariate gaussian density Properties of the gaussian density Central limit theorem Density of a transformation Higher-order statistics Kurtosis and classification of densities Cumulants, moments, and their properties Stochastic processes * Introduction and definition Stationarity, mean, and autocorrelation Wide-sense stationary processes Time averages and ergodicity Power spectrum Stochastic signal models Concluding remarks and references 51 Problems 51 3 Gradients and Optimization Methods Vector and matrix gradients Vector gradient Matrix gradient Examples of gradients 59 Copyright c 2001 John Wiley & Sons vi All rights reserved

3 3.1.4 Taylor series expansions Learning rules for unconstrained optimization Gradient descent Second-order learning The natural gradient and relative gradient Stochastic gradient descent Convergence of stochastic on-line algorithms * Learning rules for constrained optimization The Lagrange method Projection methods Concluding remarks and references 75 Problems 75 4 Estimation Theory Basic concepts Properties of estimators Method of moments Least-squares estimation Linear least-squares method Nonlinear and generalized least squares * Maximum likelihood method Bayesian estimation * Minimum mean-square error estimator Wiener filtering Maximum a posteriori (MAP) estimator Concluding remarks and references 99 Problems Information Theory Entropy Definition of entropy Entropy and coding length Differential entropy Entropy of a transformation Mutual information Definition using entropy Definition using Kullback-Leibler divergence 110 Copyright c 2001 John Wiley & Sons vii All rights reserved

4 5.3 Maximum entropy Maximum entropy distributions Maximality property of gaussian distribution Negentropy Approximation of entropy by cumulants Polynomial density expansions Using expansions for entropy approximation Approximation of entropy by nonpolynomial functions Approximating the maximum entropy Choosing the nonpolynomial functions Simple special cases Illustration Concluding remarks and references 120 Problems 120 Appendix proofs Principal Component Analysis and Whitening Principal components PCA by variance maximization PCA by minimum MSE compression Choosing the number of principal components Closed-form computation of PCA PCA by on-line learning The stochastic gradient ascent algorithm The subspace learning algorithm The PAST algorithm * PCA and back-propagation learning * Extensions of PCA to nonquadratic criteria * Factor analysis Whitening Orthogonalization Concluding remarks and references 143 Problems 144 Copyright c 2001 John Wiley & Sons viii All rights reserved

5 Part II BASIC INDEPENDENT COMPONENT ANALYSIS 7 What is Independent Component Analysis? Motivation Definition of independent component analysis ICA as estimation of a generative model Restrictions in ICA Ambiguities of ICA Centering the variables Illustration of ICA ICA is stronger that whitening Uncorrelatedness and whitening Whitening is only half ICA Why gaussian variables are forbidden Concluding remarks and references 162 Problems ICA by Maximization of Nongaussianity Nongaussian is independent Measuring nongaussianity by kurtosis Extrema give independent components Gradient algorithm using kurtosis A fast fixed-point algorithm using kurtosis Examples Measuring nongaussianity by negentropy Critique of kurtosis Negentropy as nongaussianity measure Approximating negentropy Gradient algorithm using negentropy A fast fixed-point algorithm using negentropy Estimating several independent components Constraint of uncorrelatedness Deflationary orthogonalization Symmetric orthogonalization ICA and projection pursuit Searching for interesting directions Nongaussian is interesting Concluding remarks and references 197 Copyright c 2001 John Wiley & Sons ix All rights reserved

6 Problems 198 Appendix proofs ICA by Maximum Likelihood Estimation The likelihood of the ICA model Deriving the likelihood Estimation of the densities Algorithms for maximum likelihood estimation Gradient algorithms A fast fixed-point algorithm The infomax principle Examples Concluding remarks and references 214 Problems 218 Appendix proofs ICA by Minimization of Mutual Information Defining ICA by mutual information Information-theoretic concepts Mutual information as measure of dependence Mutual information and nongaussianity Mutual information and likelihood Algorithms for minimization of mutual information Examples Concluding remarks and references 225 Problems ICA by Tensorial Methods Definition of cumulant tensor Tensor eigenvalues give independent components Tensor decomposition by a power method Joint approximate diagonalization of eigenmatrices Weighted correlation matrix approach The FOBI algorithm From FOBI to JADE Concluding remarks and references 236 Problems 236 Copyright c 2001 John Wiley & Sons x All rights reserved

7 12 ICA by Nonlinear Decorrelation and Nonlinear PCA Nonlinear correlations and independence The Hérault-Jutten algorithm The Cichocki-Unbehauen algorithm The estimating functions approach * Equivariant adaptive separation via independence Nonlinear principal components The nonlinear PCA criterion and ICA Learning rules for the nonlinear PCA criterion The nonlinear subspace rule Convergence of the nonlinear subspace rule * Nonlinear recursive least-squares rule Concluding remarks and references 261 Problems Practical Considerations Preprocessing by time filtering Why time filtering is possible Low-pass filtering High-pass filtering and innovations Optimal filtering Preprocessing by PCA Making the mixing matrix square Reducing noise and preventing overlearning How many components should be estimated? Choice of algorithm Concluding remarks and references 272 Problems Overview and Comparison of Basic ICA Methods Objective functions vs. algorithms Connections between ICA estimation principles Similarities between estimation principles Differences between estimation principles Statistically optimal nonlinearities Comparison of asymptotic variance * Comparison of robustness * Practical choice of nonlinearity 279 Copyright c 2001 John Wiley & Sons xi All rights reserved

8 14.4 Experimental comparison of ICA algorithms Experimental set-up and algorithms Results for simulated data Comparisons with real-world data References Summary of basic ICA 287 Appendix Proofs 288 Part III EXTENSIONS AND RELATED METHODS 15 Noisy ICA Definition Sensor noise vs. source noise Few noise sources Estimation of the mixing matrix Bias removal techniques Higher-order cumulant methods Maximum likelihood methods Estimation of the noise-free independent components Maximum a posteriori estimation Special case of shrinkage estimation Denoising by sparse code shrinkage Concluding remarks ICA with Overcomplete Bases Estimation of the independent components Maximum likelihood estimation The case of supergaussian components Estimation of the mixing matrix Maximizing joint likelihood Maximizing likelihood approximations Approximate estimation by quasiorthogonality Other approaches Concluding remarks 313 Copyright c 2001 John Wiley & Sons xii All rights reserved

9 17 Nonlinear ICA Nonlinear ICA and BSS The nonlinear ICA and BSS problems Existence and uniqueness of nonlinear ICA Separation of post-nonlinear mixtures Nonlinear BSS using self-organizing maps A generative topographic mapping approach * Background The modified GTM method An experiment An ensemble learning approach to nonlinear BSS Ensemble learning Model structure Computing Kullback-Leibler cost function * Learning procedure * Experimental results Other approaches Concluding remarks Methods using Time Structure Separation by autocovariances An alternative to nongaussianity Using one time lag Extension to several time lags Separation by nonstationarity of variances Using local autocorrelations Using cross-cumulants Separation principles unified Comparison of separation principles Kolmogoroff complexity as unifying framework Concluding remarks 353 Copyright c 2001 John Wiley & Sons xiii All rights reserved

10 19 Convolutive Mixtures and Blind Deconvolution Blind deconvolution Problem definition Bussgang methods Cumulant-based methods Blind deconvolution using linear ICA Blind separation of convolutive mixtures The convolutive BSS problem Reformulation as ordinary ICA Natural gradient methods Fourier transform methods Spatiotemporal decorrelation methods Other methods for convolutive mixtures Concluding remarks 368 Appendix Discrete-time filters and the -transform Other Extensions Priors on the mixing matrix Motivation for prior information Classic priors Sparse priors Spatiotemporal ICA Relaxing the independence assumption Multidimensional ICA Independent subspace analysis Topographic ICA Complex-valued data Basic concepts of complex random variables Indeterminacy of the independent components Choice of the nongaussianity measure Consistency of estimator Fixed-point algorithm Relation to independent subspaces Concluding remarks 387 Copyright c 2001 John Wiley & Sons xiv All rights reserved

11 Part IV APPLICATIONS OF ICA 21 Feature Extraction by ICA Linear representations Definition Gabor analysis Wavelets ICA and Sparse Coding Estimating ICA bases from images Image denoising by sparse code shrinkage Component statistics Remarks on windowing Denoising results Independent subspaces and topographic ICA Neurophysiological connections Concluding remarks Brain Imaging Applications Electro- and magnetoencephalography Classes of brain imaging techniques Measuring electric activity in the brain Validity of the basic ICA model Artifact identification from EEG and MEG Analysis of evoked magnetic fields ICA applied on other measurement techniques Concluding remarks Telecommunications Multiuser detection and CDMA communications CDMA signal model and ICA Estimating fading channels Minimization of complexity Channel estimation * Comparisons and discussion Blind separation of convolved CDMA mixtures * Feedback architecture Semiblind separation method Simulations and discussion 432 Copyright c 2001 John Wiley & Sons xv All rights reserved

12 23.5 Improving multiuser detection using complex ICA * Data model ICA based receivers Simulation results Concluding remarks and references Other Applications Financial applications Finding hidden factors in financial data Time series prediction by ICA Audio separation Further applications 448 References 449 Index 476 Copyright c 2001 John Wiley & Sons xvi All rights reserved