Independent Component Analysis

Transcription

1 Indeendent Comonent Analyss

2 Mture Data Data that are mngled from multle sources May not now how many sources May not now the mng mechansm Good Reresentaton Uncorrelated, nformaton-bearng comonents PCA and Fsher s lnear dscrmnant De-mng or searaton ICA Indeendent comonent analyss How do they dffer? PR, ANN, & ML

3 PCA vs. ICA Indeendent events vs. Uncorrelated events Knowng X does tell somethng about X Knowng X doesn t tell anythng about X PR, ANN, & ML 3

4 Uncorrelated vs. Indeendence Uncorrelated Global roerty Not vald under nonlnear transform PCA requres uncorrelaton Indeendence Local roerty Vald for nonlnear transform ICA assumes ndeendence ndeenden ce : E g uncorrelated : E E, g, L, g E n 0 n E g L E g n n g PR, ANN, & ML 4

5 Uncorrelated vs. Indeendence Indeendence s stronger, requrng every ossble functon of to be uncorrelated wth Ey-Eyy-Ey0 -> uncorrelated y y -> not ndeendent PR, ANN, & ML 5

6 Uncorrelated vs. Indeendence Dscrete varables X and X 0,, 0,-,,0,-,0 all wth ¼ robablty X and X are uncorrelated E 0!/4E E PR, ANN, & ML 6

7 ICA Lmtaton Any symmetrcal dstrbuton of and around orgn centered at E and E s uncorrelated Corollary: ICA does not aly to Gaussan varables Because any orthogonal transform rotaton and reflecton of Gaussan doesn t change anythng PR, ANN, & ML 7

8 Blnd Source Searaton PR, ANN, & ML 8

9 Blnd Source Searaton Bran magng Dfferent arts of bran emt sgnals that are med u n the sensors outsde the bead Teleconferencng Dfferent seaers tal at the same tme that are med u n the mcrohones Geology Ol eloraton wth underground detonaton and shoc waves beng regstered at multle sensors PR, ANN, & ML 9

10 Aroaches Nonlnear de-correlaton The de-correlated comonents are uncorrelated and the transformed de-correlated comonents are uncorrelated Mnmum mutual nformaton model Mamum non-gaussanty Mamum non-gaussanty Central lmt theorem states more Gaussanty wth successve mture Go above covarance matr urtoss, a hgherorder cumulant PR, ANN, & ML 0

11 Mathematc Formulaton s : sources, j : mtures A: mture matr W: de-mng matr Imlcaton Cannot determne the varance of sources Cannot determne the orderng of source PR, ANN, & ML

12 A Smle Formulaton Central Lmt Theorem states that sum of ndeendent random varables tends to Gaussan Non-Gaussanty s desred for each ndeendent comonent PR, ANN, & ML

13 A Smle Formulaton Gaussan varables have zero Kurtoss 4 4 urt E 3 E E 3 f E Suergaussan: sy df wth heavy tals e.g., Lalace dstrbuton e Subgaussan: flat df e.g., unform Mamze magntude of the Kurtoss e PR, ANN, & ML 3

14 urt urt a Math Framewor: varables observatons For ndeendent varables : + a 4 urt urt + urt All varables, s and y, are of unt varance Z s constraned to the unt crcle Mamum urtoss at two drectons that le n z -, z0 or z - z0 Through gradent search n w Drawbac: nose senstvty PR, ANN, & ML 4

15 Informaton Recall some mortant concets Random varable Probablty dstrbuton on a random varable Amount of nformaton, surrse, uncertanty I log log Entroy weghted, average 0 H E I I log PR, ANN, & ML 5

16 Entroy Bascs H H y I;y H[X,Y] H[Y] + H[X Y] Hy H;y Hy PR, ANN, & ML 6

17 Mutual Informaton H H y I;y Hy H;y Hy PR, ANN, & ML 7

18 D q Kullbac-Lebler dvergence log q log q Informaton dvergence, relatve entroy Measure of dfference between two dstrbutons, but t s not a metrc D q Dq D q s ostve and s zero f and only f and q have the same dstrbuton Can be a useful measurement of ndeendence, f s jont robablty q s margnal robablty Then D q s zero f and only f random varables are ndeendent,y and qy, the same as sayng that and y are ndeendent + log H, q H PR, ANN, & ML 8

19 PR, ANN, & ML 9 Intuton Indeendence mles roduct of margnal robabltes equals total robablty The Kullbac-Lebler dvergence should be mnmzed,,,,,, n n n n n n g g g g g L L L L y D y y y y log ~ y g g g g g D log ~ y y y y

20 PR, ANN, & ML 0 Math Detals A should mnmze the mutual nformaton between the new sgnal HY and the orgnal sgnal HX logdet X H Y H A X H Y H Y I AX Y X H X H X I

21 Informaton Theoretc Aroach Gaussan varable has the largest entroy among all varables of equal varance Negentroy non-gaussanalty J s to be mamzed X gauss and X have the same varance JX HX gauss -HX Dffculty: comutng H requres df Estmaton: J E urt PR, ANN, & ML

22 Mamum Entroy Aroach t d t s t A t y t Ws t y t y s s t J d t t t d t t d t PR, ANN, & ML