Independent Component Analysis. PhD Seminar Jörgen Ungh

Transcription

1 Independent Component Analysis PhD Seminar Jörgen Ungh

2 Agenda Background a motivater Independence ICA vs. PCA Gaussian data ICA theory Examples

3 Background & motivation The cocktail party problem Bla bla Bla bla Hi hi

4 Background & motivation The cocktail party problem Bla bla Bla bla Hi hi

5 Background & motivation The cocktail party problem x1 s1 Bla bla s2 s3 Bla bla Hi hi x2 x3

6 Cocktail party problem Let s1(t), s2(t) and s3(t) be the original spoken signals Let x1(t), x2(t) and x3(t) be recorded signals The connection between s and x can be written x1(t)=a11*s1(t) + a12*s2(t) + a13*s3(t) x2(t)=a21*s1(t) + a22*s2(t) + a23*s3(t) x3(t)=a31*s1(t) + a32*s2(t) + a33*s3(t) Goal: Estimate s1, s2 and s3 from x1, x2 and x3? Problem: We do not know anything about the right side

7 Cocktail party problem Example Microphone 1 Microphone 2 Separated 1 Separated 2

8 Today we celebrate our independence day - US President THOMAS J. WHITMORE (Bill Pullman) in Independence Day (1996)

9 Independence what is it?? Independence = Uncorrelation

10 Definitions Covariance: C xy = E{( x m x )( y m y ) T } Correlation: R = E{ x xy y T } If m = m = 0, then C = x y xy R xy

11 Uncorrelated Two vectors are uncorrelated if: C xy = E{( x m )( y m ) } = x y T 0 xy T R = E{ x y } = E{ x} E{ y } = T m x m T y If m x = m y = 0, then C = R = xy xy 0 from now we assume zero mean variables

12 Independent Vectors x,y are independent if: p Which also gives: ( x, y) p ( x) p ( y), y x y x = E{ g ( x) g ( y)} E{ g ( x)} E{ g ( y)} = x y Where gx and gy are arbitrary functions of x and y x y

13 Independent Independent is stronger than uncorrelated! T E { x y } = E{ x} E{ y T } E{ g ( x) g ( y)} E{ g ( x)} E{ g ( y)} = x y x y Equal if linear functions of x and y

14 Independent Uncorrelated y y x x Are x and y uncorrelated?

15 Independent Uncorrelated y y YES x YES x Are x and y uncorrelated?

16 Independent Uncorrelated y y x x Are x and y independent?

17 Independent Uncorrelated y y YES x No x Are x and y independent?

18 Relations Independent Uncorrelated BUT Uncorrelated Independent

19 ICA vs. PCA Independent Principal

20 PCA Goal: Project data onto an ortonormal basis with maximum variance Data explained by principal components e1 e2

21 PCA Uses information up to second moment, i.e. the mean and variance/covariance Reduce dimensions of data Ortonormal basis of uncorrelated vectors

22 ICA Goal: Find the independent sources Data explained by independent components y e1 x e2

23 ICA Uses information over second moment, i.e. higher order statistics like kurtosis and skewness Does not reduce dimensions of data A basis of independent vectors

24 ICA vs. PCA Independent is stronger In case of Gaussian data, ICA = PCA

25 Gaussian data

26 Gaussian distribution Definition: = ) ( ) ( 2 1 exp ) (2 1 ) ( / µ µ π x C x C x f T N x C = covariance matrix, µ = mean vector Explained completely by first and second order statistics, i.e. mean and variances

27 Gaussian data Cannot perform a rotation of the basis, due to symmetry

28 Gaussian distribution Completely defined by its first and second moment Uncorrelated Gaussian data Independence Why assume gaussian data?

29 Central limit theorem Definition: A sum of independent random variables will tend to be Gaussian That is the argument for many assumptions on gaussian distributions

30 Central limit theorem Definition: A sum of independent random variables will tend to be Gaussian What if we put it in another way?

31 Central limit theorem 2:nd definition: The mixtures of two or more independent random variables are more gaussian than the random variables themselves

32 A single random u.d. variable

33 A mixture of 2 u.d. variables

34 Idea! The observed mixtures should be more gaussian than the original components The original components should be less gaussian than the mixture If we try to maximize the non-gaussianity of the data, we should get closer to the original components

35 ICA theory Problem definition Solution Preprocessing Different methods Examples

36 ICA: Definition of the problem Let s1(t), s2(t) and s3(t) be the original signals Let x1(t), x2(t) and x3(t) be collected signals The connection between s and x can be written x1(t)=a11*s1(t) + a12*s2(t) + a13*s3(t) x2(t)=a21*s1(t) + a22*s2(t) + a23*s3(t) x3(t)=a31*s1(t) + a32*s2(t) + a33*s3(t) Goal: Estimate s1, s2 and s3 from x1, x2 and x3?

37 ICA: Assumption Independence Non-gaussian Square mixing matrix

38 ICA: Idea Maximize non-gaussianity of the data! We need a measure of Gaussianity or Non-gaussianity

39 Measures of Gaussianity 1. Kurtosis { 4} ( { 2} ) 3 2 kurt( y) = E y E y Assuming zero mean variables

40 Measures of Gaussianity 1. Kurtosis kurt( y) { } 3 = E y 4 Assuming zero mean and unit variance

41 Measures of Gaussianity 1. Kurtosis { 4} ( { 2} ) 3 2 kurt( y) = E y E y For Gaussian data we have: { } ( { }) y 4 3 E y 2 2 E = Which gives kurt(y) = 0, for Gaussian data For most other, kurt 0, positive or negative

42 Measures of Gaussianity 1. Kurtosis Maximize kurt(y)

43 Measures of Gaussianity 1. Kurtosis Maximize kurt(y) Advantages: Drawbacks: - Easy to compute - Sensitive to outliers

44 Measures of Gaussianity 2. Negentropy J ( y) = H ( ygauss ) H ( y) where, H = entropy, defined as: H ( y) = py ( η)logpy ( η) dη

45 Measures of Gaussianity 2. Negentropy J ( y) = H ( ygauss ) H ( y) Gaussian data has the largest entropy, meaning that it is the most random distrubution.

46 Measures of Gaussianity 2. Negentropy J ( y) = H ( ygauss ) H ( y) Gaussian data has the largest entropy, meaning that it is the most random distrubution. J(y) > 0 and equals zero if y gaussian

47 Measures of Gaussianity 1. Negentropy Maximize J(y) Advantages: Drawbacks: - Robust - Computationally hard

48 ICA: Solutions Kurtosis Negentropy Maximum likelihood Infomax Mutual information

49 ICA: Solutions Kurtosis Negentropy Maximum likelihood Infomax Mutual information Based on independence and/or non-gaussian

50 ICA: Restrictions Non-gaussian data* Scaling, sign and order of components Need to know the No. of components

51 ICA: Restrictions Non-gaussian data* Scaling, sign and order of components Need to know the No. of components * In case of some Gaussian data, the independent components will still be found, but the Gaussian ones will be mixed.

52 ICA: Preprocessing No reduction of dimension in ICA Need to know the number of components But, we do already have a method for dimension reduction and estimating the probable number of components

53 ICA: Preprocessing No reduction of dimension in ICA Need to know the number of components But, we do already have a method for dimension reduction and estimating the probable number of components Use PCA as a preprocessing step!

54 ICA: Preprocessing Low pass filtering + Reduces noise Reduces independence High pass filtering + Increases independence - Increases noise

55 ICA: Overlearning Much more mixtures than independent components Spiky character of the components

56 Examples Cocktail party Music separation Image analysis Separation of recorded signals of brain activity Process data Noise/Signal separation Process monitoring

57 Cocktail party problem Bla bla Bla bla Hi hi

58 Music separation Source 1 Source 2 Source 3 Source 4 Mix 1 Est 1 Mix 2 Est 2 Mix 3 Est 3 Mix 4 Est 4

59 Music separation Source 1 Source 2 Source 3 Source 4 Mix 1 Est 1 Mix 2 Est 2 Mix 3 Est 3 Mix 4 Est 4

60 Image analysis - NLPCA

61

62

63

64 Brain activity S3 S2 S1 S4

65 Process data 5 Mixed signals

66 Process data 20 Whitened signals

67 Process data 20 Independent components

68 Process data

69 Noise removal Different noise sources Laplacian Gaussian Uniform Exponential

70 Noise removal - Laplacian 4 Mixed signals

71 Noise removal - Laplacian 3 Whitened signals

72 Noise removal - Laplacian 2 Independent components

73 Noise removal - Gaussian 4 Mixed signals

74 Noise removal - Gaussian 2 Whitened signals

75 Noise removal - Gaussian 2 Independent components

76 Noise removal - Uniform 2 Mixed signals

77 Noise removal - Uniform 2 Whitened signals

78 Noise removal - Uniform 2 Independent components

79 Noise removal - Exponential 2 Mixed signals

80 Noise removal - Exponential 3 Whitened signals

81 Noise removal - Exponential 2 Independent components

82 Process monitoring Often done by PCA Example: F1, F2 One step further, use ICA!

83 Practical considerations Noise reduction (filtering) Dimension reduction (PCA?) Overlearning Algorithm

84 What about time signals? So far, no information about time used Original ICA, x is a random variable What if x is a time signal x(t)? x(t)

85 Time signal x(t) Extra information, order is not random: Autocorrelation Cross correlation More information Relaxed assumptions Gaussian data ok

86 Extensions Non-linear ICA Independent subspace analysis

87 Further information: Book: Independent Component Analysis - A. Hyvärinen, J. Karhunen, E. Oja Covers everything from novel to expert Homepage: Tutorials, material, contacts, matlab code, Journal of Machine Learning Research Papers and publications Toolboxes, code