Machine Learning. Principle Component Analysis. Prof. Dr. Volker Sperschneider

Transcription

1 Mach Larg Prcpl Compot Aalyss Prof. Dr. Volkr Sprschdr AG Maschlls Lr ud Natürlchsprachlch Systm Isttut für Iformatk chsch Fakultät Albrt-Ludgs-Uvrstät Frburg

2 I. Archtctur II. Larg - Hbb rul rloadd III. Corrlato matr: basc facts IV. Prcpl compots V. Hbb rul tratd VI. Oa s rul - rparg cogtv adquatss - VII. Sagr s rul

3 I. Archtctur

4 Archtctur Sgl lar uro y

5 II. Larg - Hbb rul rloadd

6 Larg Hbb rul rloadd Hbb larg as slf-rforcmt: { 0, } Dra put th probablty p Comput lar output y Updat ght vctor Δ

7 Larg Hbb rul rloadd Hbb larg mas slf-rforcmt: Wght from put uro ad output uro crass proportoal to actvty of uro tms actvty of output uro. hs s Hbbs rul. h strogr output s th strogr cras ghts.

8 Larg Hbb rul rloadd Do ot updat ght vctor aftr ach sgl prstato of a put vctor.ak stad of ths th ght updat avragd ovr all put vctors, that s, th pctd ght updat. [ ] [ ] [ ] Δ p p p p E E E E

9 III. Corrlato matr: basc facts

10 Corrlato matr: basc facts Epctd products of all pars of put uros form th corrlato matr C of dstrbuto p. C E p [ ] Wght updat rads as follos: Δ Δ C C,

11 Corrlato matr: basc facts Mor compact: [ ] C E p Not that s a ro vctor ad a colum vctor so that valuats to a matr othr tha r product that valuats to a scalar valu.

12 Corrlato matr: basc facts C s symmtrc: C C C s postv-dft: C E p [ ] [ ] [ E E ] p p If you ar doubt th matr multplcatos, do th sam computato plctly th sums.

13 Corrlato matr: basc facts All g-valus of C ar o-gatv: C λ th 0 mpls λ 0 sc: 0 C λ λ W smplfy th follog dscusso by assumg that all g-valus ar dffrt o multpl occurrc of a g-valu.

14 Corrlato matr: basc facts Lt all g-valus of C b umratd dcrasg odr: λ > λ > > λ 0 Cosdr assocatd g-vctors:,,, th C λ,, Normalz g-vctors to orm :,,

15 Corrlato matr: basc facts Eg-vctors to dffrt g-valus ar orthogoal: For coclud 0 C C C C C λ λ λ λ λ λ λ 0

16 Corrlato matr: basc facts Eg-vctors form a bass th th follog dcompostos of vctors: mpls k k k k k k k k

17 Corrlato matr: basc facts Eg-vctor th orm to som g-valu s uqu up to sg or - but ot that ths holds oly sc assumd to hav o multpl gvalus s dtty matr. Proof: Cosdr th C λk

18 Corrlato matr: basc facts hus: hus: k k k k k k k ± ± 0 0 λ λ λ λ k k C C C λ λ λ

19 Corrlato matr: basc facts Assum that data ar ctrd aroud zro vctor, that s, E p [ ] 0 Cosdr a arbtrary vctor th orm : Varac of data drcto s computd usg that th follog prsso s th squard lgth of th procto of oto :

20 Corrlato matr: basc facts [ ] [ ] [ ] [ ] C E E E E p p p p p σ

21 IV. Prcpl compots

22 Prcpl compots Lmma: Eprsso, for vctors of orm, taks st mamum valu for Proof: Cosdr thus th C ±

23 Prcpl compots taks ts mamum f st: th C C C λ λ thus thus ± ±

24 Prcpl compots

25 I drcto of th g-vctor th gratst g-valu varac of varac of data s mamum. If subtract from ach data vctor ts procto oto th frst prcpl drcto, gt a data dstrbuto th subspac orthogoal zu th frst prcpl compot. hs rducd data hav mamum varac drcto of th g-vctor th scod gratst g-valu.

26 V. Hbb rul tratd

27 Hbb rul tratd Hbb tratd: t t C hs s gradt dsct th rspct to th follog fucto C mag that Hbb rul attmpts to covrg to a drcto of mamum data varac. t

28 Hbb rul tratd Bad s: Hbb rul dos ot covrg ulss start th ull vctor. hs ca b s f study voluto of ght vctor dcomposd by g-vctors: 0

29 Hbb rul tratd t t t t C 0 λ λ λ λ

30 Hbb rul tratd Good s: Covrgc of drcto taks plac, that s, f ormalz vctors to, th ths ormalzd vctors covrg. Normalzd Hbb rul rads as follos: t t t C C t t

31 Hbb rul tratd From g-vctor dcomposto of 0 comput g-vctor dcomposto of C C C C th C C λ λ β β β

32 Hbb rul tratd W scal drcto vctors so that coffct of th frst g-vctor bcoms ths ca b do thout affctg drcto. Evoluto of coffcts s commo domator cacls out :

33 Hbb rul tratd,,, :,,, :,, : t t stp t stp start λ λ λ λ λ λ λ λ

34 Hbb rul tratd I th computd drcto vctors all compots th th cpto of th frst covrg to 0 ulss ufortuatly startd th a drcto th zro compot th drcto of th frst g-vctor. hus, ormalzd Hbb rul lads to covrgc to /- frst prcpl gvctor th th cpto mtod abov.

35 Hbb rul tratd A drop of bttrss: Udr ormalzato Hbb rul looss ts local charactr. Cosdr th prstato of a sgl vctor. Orgal Hbb: y th y t t t pr-syaptc uro y post-syaptc uro

36 Hbb rul tratd Normalzd Hbb: h frst part s a local updat, th scod part s global, that s, volvs all ghts to updat a sgl o. t t t t t t t u u u y th y u

37 VI. Oa s rul - rparg cogtv adquatss -

38 Oa s rul - rparg cogtv adquatss Wth actual ght vctor ad o prstato of a trag vctor, fucto g s dvlopd to a aylor srs: g g0 g' 0 Wrt domator of fucto g as follos:

39 Oa s rul - rparg cogtv adquatss Dffrtato of a quott lads to:

40 Oa s rul - rparg cogtv adquatss 3 0 ' 0 g g

41 Oa s rul - rparg cogtv adquatss If start larg th a ght vctor of orm rul smplfs to a local rul: or updatd Δ Δ

42 Oa s rul - rparg cogtv adquatss Compactly prssd usg actvato valu y: y Δ y y Frst trm s Hbb larg. Scod trm s ght dcay srvg th purpos to kp ght vctors ormd to.

43 Oa s rul - rparg cogtv adquatss Avragg dvdual larg rul Δ ovr put vctors dra by drstbuto p obta: Oa s rul Δ C C

44 VII. Sagr s rul

45 Sagr s rul Oa s rul tracts frst prcpl compots. Furthr prcpl compots ca b obtad by usg furthr output uros th latral hbto. Archtctur s llustratd th a ampl th 4 put uros ad 3 output uros:

46 Sagr s rul 3 frst prcpl compot scod prcpl compot thrd prcpl compot Not ho fd hbtory ghts ar drctd.

47 Sagr s rul h thr output uros comput: No Oa s rul s appld to tra ach uro sparatly y y y y y y

48 Frst output uro: Sagr s rul Is ot affctd by th othr uros. It fally tracts frst prcpl compot. Stadly hbts th othr to uros. Scod output uro: Ifluc of frst uro stadly lads to th subtracto of frst prcpl compot from data. hus scod uro tras by Oa s rul th subspac orthogoal to frst prcpl compot. It fally tracts scod prcpl compot.

49 Sagr s rul h follog basc ruls do th hghly otrval ob of prcpl compot tracto: - Hbb rul - Wght dcay - Latral hbto Prcpl compot tracto s a ctral mas of rdudacy rducto dmso rducto data. It may b combd th ay othr larg mchasm.