Prcpal Compoets Aalss A Method of Self Orgazed Learg Prcpal Compoets Aalss Stadard techque for data reducto statstcal patter matchg ad sgal processg Usupervsed learg: lear from examples wthout a teacher Dscover patters the put data Reles o local learg rules Utlzes Hebba learg Heavl spred b bolog 2 Basc Ituto Global order ca arse from local teractos Ala Turg The effectveess of a varable sapse betwee two euros s creased b the repeated actvato of oe euro b the other across the sapse Hebb Combe the two hpothess, ad we have a global learg strateg 3
Self Orgazed Learg Prcpals Modfcatos saptc weghts ted to self amplf Lmtato of resources leads to competto amog sapses ad therefore the selecto of the most vgorousl growg sapses at the expese of the others 4 Self Orgazed Learg Prcpals Modfcatos saptc weghts ted to cooperate Order ad structure the actvato patters represet redudat formato that s acqured b the eural etwork the form of kowledge, whch s a ecessar prerequste to self-orgazed learg 5 Self Orgazed Learg for vsual Applcatos Multple laer etworks, each laer s resposble for a certa feature Smple features e.g. cotrast, edges are hadled b earl laers, ad complex features e.g. surface texture are hadled b later laers Learg proceeds o a laer b laer bass 6
Tpcal Model Each euro acts as a lear comber Coectos are fxed throughout the learg process,.e. coectos ma be stregtheed or weakeed but ot added or destroed 7 Feature Extracto The process of trasformg a data space to a feature space of the same dmeso To ad data aalss, the trasformato t ca be from a put space of a gve dmeso to a feature space of a smaller dmeso The trasformato should be vertble ad should mmze the mea-square error of the resultg space 8 Prcpal Compoets Aalss For a put vector X: Normalze X to have a zero mea DefeamatrxR= EXX T The ormalzed Eucldea orm = 1 egevectors of the matrx R defe the drectos of maxmum varace The correspodg egevalues defe the correspodg varace values 9
Feature Mappg The egevectors of the matrx R are called the prcpal drectos. Let s call these vectors q The proecto of the put vector x oto the prcpal vector are called the prcpal compoets: a =q T x=x T q B substtutg the vector A stead of the vector x, the data s trasformed to a prcpal compoets feature space 10 PCA Illustrated 11 Dmesoalt Reducto Features ca be ordered order of domace accordg to the values of the varaces = ege values To reduce the dmesoalt of the feature space, clude the feature space the domat l compoets where l < dmesoalt of the put space The resultg error s defed as: m e = 1a q = l+ 12
Example Orgal represetato of characters the form of 32*32 pxels a vector of sze 1024. Images represet creasg dmesoalt up to 64 a reducto b a factor of 16 at rght most colum 13 Hebba Based Maxmum Ege Flter Assume a sgle lear euro as follows: x 1 w 1 1 w 2 x 2 m = w x = 1 x m w m 14 Hebba Based Maxmum Ege Flter Defe the learg rule: w + 1 = m w + η x 2 [ w + η x ] = 1 For a small learg rate, ths approxmates to: [ x w ] w + 1 = w + η 15
Hebba Based Maxmum Ege Flter We ca prove that: The varace of the output coverges to the largest egevalue of the correlato matrx R=X T X The saptc weght vector W coverges to the correspodg ormalzed ege vector 16 Hebba Based Prcpal Compoets Aalss Ca the sgle euro maxmum ege flter be exteded to the other prcpal compoets? How? 17 Hebba Based Prcpal Compoets Aalss Assume, a eural etwork formed of a sgle laer of lear euros wth umber of euros <= umber of puts: 18
Hebba Based Prcpal Compoets Aalss The outputs are defed b: m = w x = 1 Defe the learg rule: w = η x k = 1 wk k 19 Hebba Based Prcpal Compoets Aalss The egatve compoet has the effect of subtractg the prcpal compoets represeted b the prevous outputs We ca prove that: The varaces of the outputs coverge to the egevalues of the correlato matrx R=X T X descedg order The saptc weght vectors W coverge to the correspodg ormalzed ege vectors 20 Example 1 21
Example 2 22 Adaptve Prcpal Compoets Aalss Tra the etwork, oe euro at a tme We use feedback from each euro to all the euros that follow t order 23 Adaptve Prcpal Compoets Aalss For trag of euro, we use a etwork: 24
Adaptve Prcpal Compoets Aalss The output of euro s defed as: 1 a x w T T + = 25 Ad the learg rules: Ad the learg rules: ] [ 1 ] [ 1 a x a a w x w w = + + = + η η Kerel PCA Itroduces a o-lear hdde lear to covert o lear put spaces to a equvalet lear set of prcpal t 26 compoets Kerel PCA Example 27
Image Codg Strateges Compact codg: represet the mage as a reduced set of vectors desged to mmze errors, e.g. PCA Sparse dstrbuted codg: trasform redudac mage represetato to match redudac recogto vsual sstems 28