Feature Space. 4. Feature Space and Feature Extraction. Example: DNA. Example: Faces (appearance-based)

Transcription

1 Feaure Sace 4. Feaure Sace ad Feaure Exraco Alex M. Marez Hadous Hadousfor forece ECE S S2007 May roblems scece ad egeerg ca be formulaed as a PR oe. For hs, we eed o defe a feaure sace. A feaure sace s a colleco of feaures relaed o some roeres of he objec or eve uder sudy. Feaure: A dvdually measurable roery of he heomeo beg observed. Examle: DA DA sequecg ucleode order: adee (A), guae (G), cyose (C), hyme (). Each sequece s receded by a 5 marker ad eds wh a 3. E.g.: 5'-AAGCG-3'. Dscree feaure sace x,, x x x A, G, C, Examle: Faces (aearace-based) he four bases are deeced usg dffere fluoresce labels. hese are deeced ad rereseed as 'eaks' of dffere colors. I comuer vso (ad mage rocessg) may be covee o rerese mages as a sequece of xel eses. he Huma Geome Projec was fuded a may laboraores aroud he U.S. by he Dearme of Eergy (DOE), ad he aoal Isues of Healh (IH).

2 ???? Couous feaure sace: x, x, x Examle: Shae Aalyss I may alcaos of shae aalyss, such as morhomercs, bology, sychology, ad mage rocessg, 2D shaes are rereseed as feaure vecors he comlex doma. Each xel s esy defes a dmeso (feaure) of hs feaure sace. Here, we have assumed we have xels. A ew mage ca be comared o a se of samle face mages. x x 2. x,. x x x 2 or. x SC. x 2. Shf ad Scale Ivarace u[ xy, x2y 2,..., x u u uc u z CS u 2 r y ] C, Im...u Re f ( ze c CB A) c ( A)ex{ e CB * f ( z) f ( ze f ( z A) c CB ( A)ex{( ze ) ( A)ex{ z * Az}, * ) A( ze z Aze } f ( z A) )} Shercal Feaure Saces Huma Evoluo I may cases, he daa descrbg our objec s shercal (e.g., crcular). 2

3 Shercal Feaure Saces orm ad Var ormalzao Shae-based objec recogo: we would lke our algorhm o be vara o scale ad -lae roaos. = s Aearace-based recogo: brghess esy should o affec recogo. = s = sh orm ormalzao Ivara o esy sh x orm ormalzao: s x x ~ x x x ~ x x S x. S - y sh y s y Ivara o scale Varace ormalzao: ~ x x x 2. oe: Le us ow look a he case where we rerese he daa a feaure sace of dmesos he real doma, x. We wll laer see how our resuls exed o he oher feaure rereseaos. hus, uless oherwse oed, we wll assume x. Feaure Exraco I our feaure rereseao, x, we are rereseg he feaures x ad x j as orhogoal dmesos. We ca ow deerme how much correlaed each ar of dmesos are. Correlaos Lear correlaos (or co-relaos) raslaes o a lear relao bewee varables. If x ad x 2 are learly deede, we ca wre x 2 =f(x ), where f(.) s lear. Ex: If hey are o 00% correlaed, we have Lear leas-squares Our error fuco s a se of homogeous equaos: wh >. We ca rewre hese as Xa=0. Error fuco. 3

4 If rak(x)=, here s a uque soluo. Whe rak(x)>, R 2 >0. he, o mmze R 2, we eed o mmze (Xa) 2 = a X Xa. ha s: we wa o fd he dmeso R where he daa has larges varace. Le Q=X X. Ad assume. We ca he wre a Qa. symmerc, osve semdefe marx ow, le A a,, a. We have: Schur decomoso A QA. Ege decomoso Sce he colums of A defe a orhogoal bases,.e., A A=I, we ca wre QAA. hs s he well-kow egevalue decomoso equao. I he equaos above dag,,, ad we assume 0. Fg a lear fuco o X Sce, we wa o mmze R 2, we have he covarace marx o mmze (Ua) 2, we eed o fd he egevecors of X U U, where a=(a,a 2 ) s he ormal ad d he offse. hs ca be saed as Ua=0, where hs ew marx s called he covarace marx. Is geeral form s gve by Ierreg X he dagoal elemes are he varaces, c = 2. he off-dagoal are he covaraces, gve by c j = j j. he ew values j are fac he correlaos we were lookg for. Auocorrelao marx he correlao values defed above, are o o be cofused wh he auoxorrelao marx, defed as S=X X. he auocorrelao marx s decal o he covarace marx whe he daa has zero mea or has bee mea-ormalzed. 4

5 Proeres of X We oe ha X s a symmerc marx. I s also osve sem-defe,.e., x X x s greaer or equal ha 0 for all x. he equaly holds whe he samles used o esmae X le o a sub-saces of q dmesos, wh q<, or whe <. We oe ha we ca defe a ew covarace marx q dmesos. hs wll be symmerc ad osve defe for ay q=<dm(ra(x)). Rage ad ull sace he rage sace of a marx s gve by Or, he ull sace (orhogoal o he rage) s Ad, we have rak(m)=dm(ra(m)), ad Whe rak( X )<, he X s semdefe. Prcal Comoes Aalyss We ca ow fd he mos rereseave dmesos (.e., lear combao of feaures) from Σ X VV, where V v,, v, Λdag,,, 0. v s he vecor ha bes descrbed he daa. v s he h bes. 00% 99.8% 52.7% Prcal Comoes Aalyss he frs obvous dea s o rerese a - dmesoal sgal as closely as osble wh a -dmesoal feaure sace <<. We wa o fd a lear mag. X y Y We wa o mmze he mea-square error. Dscree Karhue-Loeve exaso Sce our rojeco s lear: 0. he colums ofare called bass vecors. We ow assume ha he colums ofform a orhoormal se: j 0 j herefore: y X. hus, Y s oly a orhoormal rasformao of X. 5

6 We ca rerese X wh < feaures: ˆ( ) y b. X redefed cosas he error s: X( ) X Xˆ(m) X y b ( y b ). hs omal choce s gve by he wellkow egevalue decomoso equao: X. If b 0,, he: X( ) y. o mmze he error we wa o elmae he smalles varaces or covaraces oly. Defos ad Dervaos he bass vecor s chose such ha x has maxmum varace..e.. s a vecor of dmesos; We ca wre: x j x j. j s a lear combao of he feaures of he orgal feaure-sace. 2 x s he vecor where he varace s max, cosraed o beg ucorrelaed o x. Ad so o. Aga, oe ha he quay o be. maxmzed s Sce eeds o be as large as ossble, we wll selec ha egevecor assocaed o he larges egevalue. I geeral, var[ k x] k, where k s he kh larges egevalue of Proery. Le y=bx, where B s a xq marx, q. Le y he, r( y) s maxmzed whe y q. o maxmze var[ x], we eed some addoal cosras.. ycally, we assume We ca ow maxmze, where s he Lagrage muller. Dffereag wh resec o, gves 0, I 0. I follows ha s a egevalue of. Idey marx. Comresso Frs q egevecors of. 6

7 Alcaos Examle : Face comresso Perhas he mos oular s daa comresso. Feaure exraco for rereseao: 0 egevecors Comuer vso ad mage rocessg. Aearace-based recogo. Saele magg. Rereseao of shaes (mah). Seech recogo. Gee rereseao ad recogo (geomcs). all egevecors Sascs, lguscs, bra magg, ec. Readg Examle 2: Bra magg regular words MRI (magec resoace magg) uses rado waves ad a srog magec feld raher ha x-rays o rovde clear ad dealed cures of eral orgas ad ssues. fmri (fucoal MRI) uses MRI o measure he quck, y meabolc chages ha ake lace a acve ar of he bra. haks o hs echology we ca deec he oso of he geeral areas of he bra (e.g., seech, sesao ad memory) he huma bra. fmri s a valuable resource o hyscas oo. How o comue PCA If he umber of samles s much smaller ha he ~ umber of feaures, he we comue Q X X sead. he egevecors ad egevalues of hs marx ~ / 2 ~ are relaed o Q by: e X e ~. rregular words Muldmesoal Scalg PCA s symmerc, he sese ha x x 2 x 2 x. MDS ca be formulaed for ay gve dsace measure. For examle, magg we have a ma of ces. he dsace bewee hese ces s he flyg me (whch s o symmerc). From hs, we ca cosruc a able of dsaces: D {d j }. MDS s a echque ha aems o solve he verse roblem: gve he able, fd he (ma) rereseao. 7

8 Merc MDS f (d j ) Smlar o PCA, merc MDS mmzes he recosruco error of he daa: d j (dj f (dj ))2 j where f (d j ) d j s a lear fuco. If he Eucldea dsace s used o calculae he dssmlares, d j, he merc MDS=PCA. Oher mercs: Mkowsk, Caberra, Agular, ec. Projeco Pursu omerc MDS hs s a more comlex verso of MDS. I s geerally dffcul o esmae all he arameers. Soluos usually clude evoluoary sraeges, e.g. geec algorhms ad smulag aealg. Ye aoher geeral aroach o dmesoaly reduco. Searches for hose dmesos ha have he mos e r e s g r o j e c o. PCA Comuaoally exesve. Exloraory PP searches hose feaures where he daa devaes PP from Gaussa. PPC, PPR, ec. Projeco Pursu Regresso Covergece Searches for hose low-dmesoal rojecos ha maxmze (mmze) a objecve fuco called rojeco dex. PCA s a arcular case of PP (whe he daa varace alog he chose rojeco s seleced as he rojeco dex). E.g., mmze he error fuco feaure vecor (u) arge M (ouu) err [ y g m ( wm x )] quas-ewo aroach): g ( w x ) g ( w x ) g ' ( w x )( w w ) x, m Rdge fuco. over he fuco g m ad dreco vecor wm. We seek wm so ha he model fs he daa correcly, hece he ame Projeco Pursu. Selec a al w, =0. Calculae v=wx. Aroxmae g() usg a sle (-surface). Wh hs ew g(), esmae he ew w (usg a err y g (w 2 2 y g ( w x ) x ) g ' ( w x ) w x g ' ( w x ) w x. 5. Go back o se 2; ul err s small (cov.) Mmze wh LS. 8

9 Ideede Comoe Aalyss Isead of mmzg he error, ICA seeks o fd deede comoes of he daa. x ( ) s our se of deede sources: d [x( )] [ x ( )] x s o observable, bu we ca assume: s( ) Ax( ). he goal s o fd a se y ( ) x ( ). x (ad, herefore, y)m g hbeas e o f a b s r a c (robably ukow) sources (comoes). We ca model he rasformao from s o y as: Oe such aroach s o use he eroy: Our goal ow s o esmae W ad w 0 so as o make he comoes of y deede from each oher. here s o close form soluo o hs roblem. We ca, however, buld a erave mehod; whch case our goal s o fd a W ha roduces a se of deede sources: Le be a loss fuco wh arge W A ad: he: whch s based o he observao ha o maxmze he eroy we eed o maxmze he muual formao bewee u ad ouu. FasICA uses a dffere fuco: y f [ Ws w 0 ]. (x) l ( x) dx here G() s a o-quadrac fuco, v s Gaussa ad c s a cosa (c>0). Examle : Bld source searao s ( ) x( ) 9

10 Examle 2: Image refleco y ( ) Examle 3: Bra erface 0