Data Modeling using Kernels and Information Theoretic Learning

Transcription

1 Data Modlg usg Krls ad Iformato Thortc Larg Jos C. Prcp Computatoal uroegrg Laboratory Elctrcal ad Computr Egrg Dpartmt Uvrsty of Florda

2 Ackowldgmts Dr. Joh Fshr Dr. Dog Xu Dr. K Hld Dr. Dz Erdogmus My studts: Puskal Pokharl Wfg Lu Jawu Xu Kyu-Hwa Jog Sudhr Rao Sugju Ha SF ECS ad 0607 urogrg program ad DARPA

3 Outl Iformato Thortc Larg Krl Mthods Corrtropy as Gralzd Corrlato Applcatos Matchd fltrg Wr fltrg ad Rgrsso olar PCA

4 Twty Yars ago ural tworks wr a paradgm shft data aalyss bcaus: Thy hadld hghr dmsoal problms tha t was cosdrd possbl Thy took advatag of olar mult-layr topologs that mplmtd vry ffct, uvrsal appromators. Thy could b trad wth a w larg algorthm calld backpropagato Thy wr may rspcts a mtaphor for bra procssg du to thr dstrbutd, olar, dyamcal atur. Howvr, thy wr mostly capturg th scod ordr statstcs of th rror bcaus of th Ma Squar Error cost fucto

5 Iformato Fltrg: From Data to Iformato Iformato Fltrs: Gv data pars {,d } Optmal Adaptv systms Iformato masurs f,w Data d Data Adaptv Systm Output rror Iformato Masur Iformato Embd formato th wghts of th adaptv systm Mor formally, us optmzato to prform Baysa stmato

6 Iformato Fltrg Dz Erdogmus ad Jos Prcp IEEE SP MAGAZIE From Lar Adaptv Fltrg to olar Iformato Procssg ovmbr 006

7 What s Iformato Thortc Larg? ITL s a mthodology to adapt lar or olar systms usg crtra basd o th formato dscrptors of tropy ad dvrgc. Ctr pc s a o-paramtrc stmator for tropy that: Dos ot rqur a plct stmato of pdf Uss th Parz wdow mthod whch s kow to b cosstt ad ffct Estmator s smooth Radly tgratd covtoal gradt dsct larg Provds a lk to Krl larg ad SVMs. Allows a tso to radom procsss

8 ITL s a dffrt way of thkg about data quatfcato Momt pasos, partcular Scod Ordr momts ar stll today th workhors of statstcs. W automatcally traslat dp cocpts.g. smlarty, Hbb s postulat of larg d ordr statstcal quvalts. ITL rplacs d ordr momts wth a gomtrc statstcal trprtato of data probablty spacs. Varac by Etropy Corrlato by Corrtopy Ma squar rror MSE by Mmum rror tropy MEE Dstacs data spac by dstacs probablty spacs

9 Iformato Thortc Larg Etropy ot all radom varabls r.v. ar qually radom! f 0.5 f Etropy quatfs th dgr of ucrtaty a r.v. Claud Shao dfd tropy as H S X px log px H S X f X log f X d August, 007 9

10 Iformato Thortc Larg Ry s Etropy H α X log α p α X H α X log α f α X d Ry s tropy quals Shao s as α orm of th pdf: p α p k k p α α tropy α-orm α orm of p rasd powr to α p p p, p, p 3 α orm V α α 0 p p, p p p 0 p 3 V α fy α dy August, 007 0

11 Iformato Thortc Larg Parz wdowg Gv oly sampls draw from a dstrbuto: { pˆ,..., } ~ p G σ Covrgc: Krl fucto lm pˆ p * Gσ provdd that σ f f f f

12 Iformato Thortc Larg Ry s Quadratc Etropy Ordr- tropy & Gaussa krls: H pˆ X log p log log d log j j G G σ j j G G Iformato pottal, V X σ σ σ d d Parws tractos btw sampls O provds a pottal fld ovr th spac of th sampls paramtrzd by th krl sz σ Prcp, Fshr, Xu, Usuprvsd Adaptv Fltrg, S. Hayk, Wly, 000.

13 Iformato Thortc Larg Iformato Forc I adaptato, sampls bcom formato partcls that tract through formato forcs. Iformato pottal: V X G j σ Iformato forc: V j j G σ j j Prcp, Fshr, Xu, Usuprvsd Adaptv Fltrg, S. Hayk, Wly, 000. Erdogmus, Prcp, Hld, atural Computg, 00.

14 Iformato Thortc Larg Backpropagato of Iformato Forcs Iput Adaptv Dsrd IT Systm Output Crtro Wght Updats Adjot twork Iformato Forcs k J J p w j p p w j Iformato forcs bcom th jctd rror to th dual or adjot twork that dtrms th wght updats for adaptato.

15 Iformato Thortc Larg Quadratc dvrgc masurs Kulback-Lblr Dvrgc: p D KL X ; Y p log d q Ry s Dvrgc: D α X ; Y α p log p α d q Euclda Dstac: Cauchy- Schwartz Dstac : p q Mutual Iformato s a spcal cas dvrgc btw th jot ad th product of margals D E X ; Y d D C X ; Y log p p q d d q d

16 Iformato Thortc Larg Ufyg crtro for larg from sampls Dsrd Sgal D Iput Sgal Larg Systm Y q X, W Output Sgal Optmzato Iformato Masur IYD, X Y

17 Trag ADALIE sampl by sampl Stochastc formato gradt SIG Batch gradt for ay α ad ay krl: Stochastc appromato SIG wth M sampls: For α ad Gaussa krl For M, th SIG bcoms LMS quvalt Erdogmus, Prcp, Hld, IEEE Sgal Procssg Lttrs, 003. ˆ M k k M k M w V σ α σ α α κ κ α ˆ M k k k G M w V σ α ˆ G w V k k k σ α j j j j w y w y w V ˆ σ α σ α α κ κ α

18 Trag ADALIE sampl by sampl Stochastc formato gradt SIG Thorm: Th pctd valu of th stochastc formato gradt SIG, s th gradt of Shao s tropy stmatd from th sampls usg Parz wdowg. For th Gaussa krl ad M Th form s th sam as for LMS cpt that tropy larg works wth dffrcs sampls. Th SIG works mplctly wth th L orm of th rror. Erdogmus, Prcp, Hld, IEEE Sgal Procssg Lttrs, 003., log ˆ L S L E H σ κ, ˆ L L k k k s E w H σ σ κ κ ˆ, w H k k k S σ

19 SIG Hbba updats Δw ηy I a lar twork th Hbba updat s k k k Th updat mamzg Shao output tropy wth th SIG bcoms Hα y w Whch s mor powrful ad bologcally plausbl? k σ y y k k k k Gratd 50 sampls of a D dstrbuto whr th as s uform ad th y as s Gaussa ad th sampl covarac matr s Erdogmus, Prcp, Hld, IEEE Sgal Procssg Lttrs, 003. Hbba updats would covrg to ay drcto but SIG foud cossttly th 90 dgr drcto!

20 Trag ADALIE sampl by sampl A MLP ca b trad layr by layr, wthout backpropagatg th rrors by mamzg th mutual formato QMI CS btw th output of ach layr ad th dsrd rspos. Ths s smlar to Lskr s IfoMa but oparamtrc ad wth a dsrd sgal. Wghts ar trad wth th dlta rul. Frqucy doublr twork D Frst Hdd od Scod Hdd od X Y Y k ma{ I Y, D } ma{ I Y k, D } td 5-- Plot th output of two hdd ods togthr Th output of th twork

21 ITL - Applcatos Systm dtfcato Fatur tracto ITL Bld sourc sparato Clustrg ITL has ampls ad Matlab cod

22 Rproducg Krl Hlbrt Spacs as a Tool for olar Systm Aalyss

23 Fudamtals of Krl Mthods Krl mthods ar a vry mportat class of algorthms for olar optmal sgal procssg ad mach larg. Effctvly thy ar shallow o layr ural tworks RBFs. Thy plot th lar structur of Rproducg Krl Hlbrt Spacs RKHS wth vry ffct computato. AY! SP algorthm prssd trms of r products has prcpl a quvalt rprstato a RKHS, ad may corrspod to a olar oprato th put spac. Solutos may b aalytc stad of adaptv, wh th lar structur s usd.

24 Fudamtals of Krl Mthods Dfto A Hlbrt spac s a spac of fuctos f. Gv a cotuous, symmtrc, postv-dft krl, a mappg Φ, ad a r product κ :U U R A RKHS H s th closur of th spa of all Φu. Rproducg Krl trck f, Φ u f u H Φ u, Φ u κ u, u H, H Th ducd orm H, f f f H

25 Fudamtals of Krl Mthods RKHS proposd by Parz For a stochastc procss {X t, tєt} wth T bg a d st, th auto-corrlato fucto R X : T T R dfd as R X t,s E[X t X s ] s symmtrc ad postv-dft, thus ducs a RKHS by th Moor-Aroszaj thorm, dotd as PRKHS PH. Furthr thr s a krl mappg Θ such that R t,s< Θt, Θs> PH Scod ordr statstcs o r. p. ar quvalt to algbrac opratos th PRKHS. PRKHS brought udrstadg but o w rsults.

26 Fudamtals of Krl Mthods RKHS ducd by th Gaussa krl Th Gaussa krl s symmtrc ad postv dft k σ ', ' p. πσ σ thus ducs a RKHS o a sampl st {, } of rals, dotd as GRKHS or GH. Furthr, by Mrcr s thorm, a krl mappg Φ ca b costructd whch trasforms data from th put spac to GRKHS whr: kσ < Φ, Φ > GH whr <,> dots r product GRKHS.

27 ITL ad Krl Mthods: Ctral momts fatur spac ITL s also a krl mthod. Iformato pottal: G V X G j σ σ j j Substtutg th formato pottal T j Φ ΛΦ < Φ j, Φ V T X Φ ˆ j Λ Φ μφ j > So Ry s quadratc tropy s th log of th orm squar of th projctd data ma,.. a ctral momt fatur spac

28 ITL ad Krl Mthods: Ctral Momts Estmators Ma: Varac E E[ X ] [ X E[ X ] ] Iformato Pottal E[ f ] G σ j j σ scal Ry s Quadratc Etropy log E[ f ] log G j σ j

29 ITL ad Krl Mthods: SVM as a ITL cost fucto Th SVM classfr mamzs subjct Rwrt as Whr,,, j j j j k d d L α α α d 0,, α α A j j * α α ˆ ˆ,,,,,,,,,, ' ' ' ' ' ' ' ' ' q p D A A k A k A k A A A L ED + α α α α α α *, ˆ, ˆ j j j k A q k A p α α

30 Adaptv Fltrg RKHS Th Krl LMS algorthm Wdrow s famd LMS algorthm ca b wrtt asly RKHS 0, 0 L t a t t a a a mp w w y w w w w d w w d R m > < + η η η, 0 0 Ω > Φ < Φ Φ Ω Φ Ω Φ + Ω Ω Φ Ω Ω Ω Φ t a t f t a t t a a a mp G y y d d R m η η η η Puskal Pokharl, Wfg Lu, Jos Prcp, 'Krl LMS', Proc. ICASSP 007, Hawa

31 Adaptv Fltrg RKHS Th Krl LMS algorthm Th KLMS gvs rs to a RBF twork that ca b trad o-l, whr th wghts ar th rrors ad t s costatly growg wth ach sampl. W hav rctly provd that th KLMS s wll posd th ss of Hadamar, so t dos ot d rgularzato as most krl algorthms th ltratur. 0. Prdcto of Macky-Glass Tm srs t ms lar ms krl η Lu W., Pokarl P., Prcp J., Th Krl LMS Algorthm, accptd IEEE Tras. Sgal Procssg.

32 Corrtropy: A w gralzd smlarty masur Corrlato s o of th most wdly usd fuctos sgal procssg. But, corrlato oly quatfs smlarty fully f th radom varabls ar Gaussa dstrbutd. Ca w df a w fucto that masurs smlarty but t s ot rstrctd to scod ordr statstcs? Us th ITL framwork.

33 Corrtropy: A w gralzd smlarty masur Df corrtropy of a radom procss { t } as V t, s E δ δ p d t s t s W ca asly stmat corrtropy usg krls Vˆ t, s E k Th am corrtropy coms from th fact that th avrag ovr th lags or th dmsos s th formato pottal th argumt of Ry s tropy For strctly statoary ad rgodc r. p. Vˆ m k m Satamara I., Pokharl P., Prcp J., Gralzd Corrlato Fucto: Dfto, Proprts ad Applcato to Bld Equalzato, IEEE Tras. Sgal Proc. vol 54, o 6, pp 87-86, 006. t s

34 Corrtropy: A w gralzd smlarty masur How dos t look lk? Th swav

35 Corrtropy: A w gralzd smlarty masur Proprts of Corrtropy: It has a mamum at th org / πσ It s a symmtrc postv fucto Its ma valu s th formato pottal Corrtropy cluds hghr ordr momts of data V s, t 0 σ E! Th matr whos lmts ar th corrtopy at dffrt lags s Topltz s t

36 Corrtropy: A w gralzd smlarty masur Corrtropy as a cost fucto vrsus MSE. MSE X Y E X Y, [ ] y f y ddy XY, y, fe d V X, Y E[ k X Y] k yf yddy, y, XY k f d E

37 Corrtropy: A w gralzd smlarty masur Corrtropy ducs a mtrc CIM th sampl spac dfd by CIM X, Y V 0,0 V X, Y / Thrfor corrtropy ca b usd as a altratv smlarty crtro th spac of sampls Lu W., Pokharl P., Prcp J., Corrtropy: Proprts ad Applcatos o Gaussa Sgal Procssg, accptd IEEE Tras. Sg. Proc

38 Corrtropy: A w gralzd smlarty masur Corrtropy crtro mplmts M stmato of robust statstcs. M stmato s a gralzd mamum lklhood mthod. I adaptato th wghtd squar problm s dfd as Wh arg θ m lm w θ ρ, θ p / / w ρ '/ ρ σ πσ ths lads to mamzg th corrtropy of th rror at th org. ψ, ˆ θ 0 m m p / / θ θ σ πσ κσ θ θ ρ σ πσ ma p / / ma M ψ ρ

39 Corrtropy: A w gralzd smlarty masur olar rgrsso wth outlrs Mddlto modl Polyomal appromator

40 RKHS ducd by Corrtropy Dfto For a stochastc procss {X t, tєt} wth T bg a d st, corrtropy dfd as s symmtrc ad postv dft. V X t,s E[KX t X s ] Thus t ducs a w RKHS dotd as VRKHS VH. Thr s a krl mappg Ψ such that V t,s< Ψt,Ψs > VH Ay symmtrc o-gatv krl s th covarac krl of a radom fucto ad vc vrsa Parz. Thrfor, gv a radom fucto {X t, t є T} thr sts aothr radom fucto {f t, t є T} such that E[f t f s ]V X t,s

41 RKHS ducd by Corrtropy Rlato wth PRKHS Corrtropy stmats smlarty fatur spac. It trasforms th compot ws data to fatur spac by f. ad th taks th corrlato V, y E[ k, y] E[ f. f y] f. s a fucto also mplctly dfd as φ was VRKHS s olarly rlatd to th put spac ulk PRKHS. VRKHS sms vry approprat to EXTED tradtoal lar sgal procssg mthods to olar systm dtfcato thr s o d for rgularzato ad tm cosumg quadratc programmg.

42 RKHS ducd by Corrtropy Rlato wth GRKHS. Gram matr stmats parws valuatos fatur spac. It s a sampl by sampl valuato of fuctos Gram matr s data sz Corrtropy matr s LL spac sz Ths mas hug savgs: If w hav a mllo sampls a 55 put spac, Gram matr s Th corrtropy matr s 55, but ach lmt s computd from a mllo pars. vry much lk a MA modl of ordr 5 stmatd wth o mllo sampls

43 Applcatos of VRKHS Matchd Fltrg Matchd fltr computs th r product btw th rcvd sgal r ad th tmplat s R sr 0. Th Corrtropy MF computs V rs 0 k r s Hypothss rcvd sgal Smlarty valu H 0 r k k V0 πσ H r k sk + k V πσ σ s / /σ Patt pdg

44 Applcatos of VRKHS Matchd Fltrg Lar Chals Wht Gaussa os Impulsv os Tmplat: bary squc of lgth 0. krl sz usg Slvrma s rul.

45 Applcatos of VRKHS Matchd Fltrg Alpha stabl os α., ad th ffct of krl sz Tmplat: bary squc of lgth 0. krl sz usg Slvrma s rul.

46 Applcatos of VRKHS Mmum Avrag Corrlato Ergy fltrs W cosdr a -dmsoal mag data as a d colum vctor. Th covtoal MACE fltr s formulatd th frq. doma. Trag Imag matr : Corrlato rgy of th -th mag s a dagoal matr whos lmts ar th magtud squard of th assocatd lmt of, that s, th powr spctrum of Corrlato valu at th org Avrag rgy ovr all trag mags MACE fltr mmzs th avrag rgy whl satsfyg th costrat

47 Applcato of VRKS Th Corrtropy MACE fltr Eplotg th lar spac proprts of th VRKHS, th optmzato problm th fatur spac bcoms Soluto Do ot kow F plctly! But ca comput Tst output,

48 Applcatos of VRKHS VMACE Smulato Rsults Fac Rcogto MACE vs. Corrtropy MACE W usd th facal prsso databas from th Advacd Multmda Procssg Lab CMU. Th databas cossts of 3 subjcts, whos facal mags wr capturd wth 75 varyg prssos. Th sz of ach mag s Tru class mags Fals class mags wth addtv Gaussa os SR0db. Rport th rsults of th two most dffcult cass who producd th worst prformac wth th MACE mthod. Th smulato rsults hav b obtad by avragg Mot-Carlo approach ovr 00 dffrt trag sts ach trag st cossts of radomly chos 5 mags Th krl sz s ~ 40% of th stadard dvato of th put data.

49 Applcatos of VRKHS VMACE ROC curvs wth dffrt SRs

50 Applcatos of VRKHS CMACE o SAR/ATR aspct agl msmatch

51 Applcatos of Corrtropy Wr Fltrg W ca also formulat th Wr fltr as a Corrtropy fltr stad of SVM rgrsso by drctly plotg th lar structur of th VRKHS. Howvr, thr s stll a appromato th computato. T y F Ω F V d k F k T k j 0 0 k L L f aj f k j d k L L dk aj{ f f k j} k j 0 0 L L dk ak j, k j k j 0 0 Patt Pdg

52 Applcatos of Corrtropy Wr Fltrg Prdcto of th Macky Glass quato a mld chaotc systm Wr vrsus corrtropy RBF 50 vrsus corrtropy RBF EW METHOD MSE DIMESIO OF PREDICTOR

53 Applcatos of VRKHS olar tmporal PCA Th Karhu Lov trasform prforms Prcpal Compot Aalyss TPCA of th autocorrlato of th r. p. R XX T... L r0 r L r L r O O M M O r0 r rl L r r 0 X L L L K L T X X r0 r L r r L O O M M O r0 r r L r r 0 T T T R XX UDD U T T T K X X VD DV D s L dagoal { λ, λ,... λ L } T T U X λ V,,,..., L TPCA ca b also do by dcomposg th Gram matr K drctly.

54 Applcatos of VRKHS olar tmporal PCA Sc th autocorrlato fucto of th projctd data VRKHS s gv by corrtropy, w ca drctly costruct K wth th corrtropy. Eampl: m As π fm zm whr + p 0.8 0, , , FFT of cla sgal ad osy obsrvato Obsrvato cla sgal FFT of frst PC usg Auto-corrlato Matr A VPCA d PC PCA by -by- 56 PCA by L-by-L L % 5% 3% 8% PCA by L-by-L L FFT of d PC usg Corrtropy Matr % 7% 6% 7% % 99% 47% 90%,000 Mot Carlo rus. σ

55 Applcatos of VRKHS Corrtropy Spctral Dsty CSD s a fucto of th krl sz, ad shows th dffrc btw PSD σ larg ad th w spctral masur Avrag ormalzd ampltud Tm Badwdth product frqucy Krl sz Krl sz

56 Applcatos of VRKHS Corrtropy basd corrlograms Corrtropy ca b usd computatoal audtory sc aalyss CASA, provdg much bttr frqucy rsoluto. Fgurs show th corrlogram from a 30 chal cochla modl for o ptch00hz. Auto-corrlato Fucto Auto-corrtropy Fucto lags

57 Applcatos of VRKHS Corrtropy basd corrlograms Corrtropy ca b usd computatoal audtory sc aalyss CASA, provdg much bttr frqucy rsoluto. Fgurs show th corrlogram from a 30 chal cochla modl for two suprmposd vowls from two spakrs f 0 00, 6 Hz. Auto-corrlato Fucto Auto-corrtropy Fucto lags

58 Coclusos Iformato Thortc Larg took us out of th local mmum of Gaussa statstcs ad MSE as cost fuctos. ITL gralzs may of th statstcal cocpts w tak for gratd. Krl mthods mplmt shallow ural tworks RBFs ad td asly th lar algorthms w all kow. KLMS s a smpl algorthm for o-l larg of olar systms Corrtropy dfs a w RKHS that sms to b vry approprat for olar systm dtfcato ad robust cotrol Corrtropy may tak us out of th local mmum of th adaptv dsg of optmum lar systms For mor formato go to th wbst ITL rsourc for tutoral, dmos ad dowloadabl MATLAB cod

59 ITL Applcatos olar systm dtfcato Mmz formato cott of th rsdual rror Dsrd, d Iput, Adaptv Systm Output, y Error, Mmum Etropy Equvaltly provds th bst dsty matchg btw th output ad th dsrd sgals. m log w α p α ε; w dε m w p y ξ, η; w p y p ξ, η; w ξ, η d α dξdη Erdogmus, Prcp, IEEE Tras. Sgal Procssg, 00. IEEE SP 003 Youg Author Award

60 ITL Applcatos Tm-srs prdcto Chaotc Macky-Glass MG-30 srs Compar crtra: Mmum squard-rror Mmum rror tropy Probablty Dsty Error Valu Systm: TD 6+4+ MEE MSE Probablty Dsty Sgal Valu Data MEE MSE

61 ITL - Applcatos Systm dtfcato Fatur tracto ITL Bld sourc sparato Clustrg

62 ITL Applcatos Optmal fatur tracto Data procssg qualty: Mutual formato s mootocally ocrasg. Classfcato rror qualty: Error probablty s boudd from blow ad abov by th mutual formato. Class labls d z Mamz Mutual Iformato z Classfr d y Compar Backpropagat Iformato Forcs PhD o fatur tracto for soar targt rcogto 00 Prcp, Fshr, Xu, Usuprvsd Adaptv Fltrg, S. Hayk, Wly, 000.

63 ITL Applcatos Etract olar faturs 6464 SAR mags of 3 vhcls: BMP, BTR70, T7 Iformato forcs trag Classfcato rsults PCorrct MI+LR 94.89% SVM 94.60% Tmplats 90.40% Zhao, Xu ad Prcp, SPIE Automatc Targt Rcogto, 999. Hld, Erdogmus, Prcp, IJC Tutoral o ITL, 003.

64 ITL - Applcatos Fatur tracto Systm dtfcato ITL Bld sourc sparato Clustrg

65 ITL Applcatos Idpdt compot aalyss Obsrvatos ar gratd by a ukow mtur of statstcally dpdt ukow sourcs. k Hs k I z H zc c H z Idpdt sourcs s y z H Sphrg W R Idpdt sgals Mmz Mutual Iformato Md sgals Ucorrlatd sgals K Hld II, PhD o bld sourc sparato 003

66 ITL Applcatos O-l sparato of md souds Off-l sparato, 00 mtur O-l sparato, 33 mtur MRMaId Ifoma 5 0 MRMaId-SIG SIR db Como FastICA SIR db Ifoma FastICA 0 0 Como Data Lgth sampls Obsrvd mturs ad sparatd outputs X: X: X3: Z: Z: Z3: Tm thousads of data sampls Hld, Erdogmus, Prcp, IEEE Sgal Procssg Lttrs, 00.

67 ITL - Applcatos Fatur tracto Systm dtfcato ITL Bld sourc sparato Clustrg

68 ITL Applcatos Iformato thortc clustrg Slct clustrs basd o tropy ad dvrgc: Mmz wth clustr tropy Mamz btw clustr dvrgc Btw clustr dvrgc m m p p q d d q d Mmbrshp vctor Wth clustr tropy / Robrt Jss PhD o formato thortc clustrg Jss, Erdogmus, Hld, Prcp, Eltoft, IEEE Tras. Pattr Aalyss ad Mach Itllgc, 005. submttd