The Histogram. Non-parametric Density Estimation. Non-parametric Approaches

Transcription

1 The Hogram Chaper 4 No-paramerc Techque Kerel Pare Wdow Dey Emao Neare Neghbor Rule Approach Neare Neghbor Emao Mmum/Mamum Dace Clafcao No-paramerc Approache A poeal problem wh he paramerc approache The uderlyg cla-codoal paer drbuo are requred boh form ad parameer Uforuaely, pracce, wo problem may are:. A pecfc form e.g. Gaua or Uform ca deerme.. The choe form doe o f oe of he emable formulao For hee reao, hey aemp o emae he dey drecly from he daa whou mag ay paramerc aumpo abou he uderlyg drbuo. No-paramerc learg echque eeded a he followg: Emao of dey fuco p ω o-paramercally drecly Drecly emae o-paramercally pω Traform he feaure pace The mple form of o-paramerc D.E. he famlar hogram Dvde he ample pace o a umber of b ad appromae he dey a he ceer of each b by he fraco of po he rag daa fall o he correpodg b [umber of ame b a ] P H N [wdh of b coag ] The hogram requre wo parameer o be defed: b wdh ad arg poo of he b The hogram a very mple form of D.E., bu ha everal drawbac The fal hape of he dey emae deped o he arg poo of he b For mulvarae daa, he fal hape of he dey alo affeced by he oreao of he b The dcoue of he emae are o due o he uderlyg dey, hey are oly a arfac of he choe b locao Thee dcoue mae very dffcul, whou eperece, o grap he rucure of he daa A much more erou problem he crue of dmeoaly, ce he umber of b grow epoeally wh he umber of dmeo I hgh dmeo we would requre a very large umber of eample or ele mo of he b would be empy All hee drawbac mae he hogram uuable for mo praccal applcao ecep for rapd vualao of reul oe or wo dmeo Therefore, we wll o ped more me loog a he hogram No-paramerc Dey Emao The probably ha a vecor, draw from a drbuo p, wll fall a gve rego R of he ample pace P R p ' d' P R Aume ow ha vecor {,,, } are depedely draw from he drbuo he probably dey fuco p ha characere cla w, are avalable rag e D The probably ha of hee vecor fall R gve by he bomal drbuo: P of vecor R P P The epeced umber of ample of he fallg R, he epeced value, whch he cae of he bomal drbuo E { } P ar{ } P P 3 4

2 No-paramerc Dey Emao Therefore, a 8, he drbuo become harper he varace ge maller o we ca epec ha a good emae of he probably P ca be obaed from he mea fraco of he po ha fall wh R P The probably P of fdg paer for large, he bomal drbuo pea rogly a /P No-paramerc Dey Emao I pracce, he value of he oal umber of eample fed I order o mprove he accuracy of he emae p we could le approach ero bu he he rego R would become o mall ha would ecloe o eample Th mea ha we wll have o fd a comprome value for he volume Large eough o clude eough eample wh R Small eough o uppor he aumpo ha p coa wh R I cocluo, he geeral epreo for o-paramerc dey emao become / p he volume urroudg where he oalumber of eample he umber of eample de O h ba, he oberved umber of vecor fallg o R he mea..e., ob P Th mea a emaor for P P ob / 5 7 No-paramerc Dey Emao Aumg ha p coa over R aume ha R o mall ha p doe o vary apprecably wh, we appromae a P R p ' d' p R where, a po wh R. he volume ecloed by R. For cla w, ug rag e D coag ample, he emaed p.d.f. P p ' d' p R P / / p.e., of he ample fall o rego R wh volume No-paramerc Dey Emao Whe applyg h reul o praccal dey emao problem, wo bac approache ca be adoped. Kerel Dey Emao Pare Wdow mehod o chooe a fed value of he volume ad deerme from he daa. o hr a al rego by pecfyg he volume a ome fuco of, uch a /qr. The ad / behave properly o he po. Neare Neghbor mehod-nn o chooe a fed value of ad deerme he correpodg volume from he daa. o epad a rego by pecfy a ome fuco of, uch a qr /qr Th emae become more accurae a we creae he umber of ample po ad hr he volume qr 6 8

3 Hypercube Fuco Aumg ha he rego R ha ecloe he eample d-dmeoal hypercube wh de of legh h ceered a he emao po. The, volume gve by h d, where d he umber of dmeo. To fd he umber of eample ha fall wh h rego we defe a erel fuco φ a d-dmeoal hypercube fuco d,,,..., φ 0 oherwe Th erel, whch correpod o a u hypercube ceered a he org, ow a a Pare wdow. φ d R d Noce ha φ φ h a ralaed ad caled vero of φ, where he orgceer of φ a Numerc Eerce Gve he daae below, ue Pare wdow o emae he dey p a y3,0,5. Ue a badwdh of h4 X {,,, } {4, 5, 5, 6,, 4, 5, 5, 6, 7} Soluo Le fr draw he daae o ge a dea of wha umercal reul we hould epec Le ow emae py3: y p y 3 d φ h h φ + φ + φ + φ + L + φ [ φ / 4 + φ / + φ / + φ 3/ 4 + L+ φ 3/ 4 ] 0 4 [ ] Smlarly p y 0 [ ] p y 5 [ ] Pare Wdow For a rag e wh ample, he umber of ample,, he hypercube rego ceered a φ h where, φ hypercube wdow fuco A geeral erpreao of u ep fuco φ a a erpolao fuco The emae oba / p Th equao ugge ha φ. may be vewed a a erpolao fuco Pare wdowfor p, where ample are avalable Noce ha he Pare wdow dey emae reemble he hogram, wh he ecepo ha he b locao are deermed by he daa po φ h f- f- f- 3 f- 4 0 Pare Wdow The Pare wdow ha everal drawbac Yeld dey emae ha have dcoue. Wegh equally all he po, regardle of her dace o he emao po I eay o overcome ome of hee dffcule by geeralg he Pare wdow wh a mooh erel fuco f whch whch afe he codo φ d R d Uually, bu o alway, φ wll be a radally ymmerc ad umodal probably dey fuco, uch a he mulvarae Gaua dey fuco T φ ep d / π where he epreo of he dey emae rema he ame a wh Pare wdow p φ h d h 0

4 Smooh Kerel Eample of Pare Wdow Ju a he Pare wdow emae ca be codered a um of boe ceered a he obervao, he mooh erel emae a um of bump placed a he daa po The erel fuco deerme he hape of he bump The parameer h, alo called he moohg parameer or badwdh, deerme her wdh -D crcularly ymmerc ormal Pare wdow φ/h for value of h0.. h0.5. h.0. δ are ormaled 3 5 Smooh Kerel Emae Pare wdow emae p of a bmodal drbuo Eample of Pare Wdow Emao Pare-wdow dey p emae baed o he e of 5 ample h0. h0.5 h.0 4 6

5 Eample of Pare Wdow Eample of Pare Wdow -D crcularly ymmerc ormal Pare wdow φ/h for value of h0.. δ are ormaled Pare-wdow dey emae baed o he e of 5 ample -D crcularly ymmerc ormal Pare wdow φ/h for value of h.0. δ are ormaled Pare-wdow dey p emae baed o he e of 5 ample 7 9 Eample of Pare Wdow Pare Wdow Emao Pare wdow emae p of a uvarae ormal dey o. of ample Wdow wdh h.0 h0.5 h0. -D crcularly ymmerc ormal Pare wdow φ/h for value of h0.5. δ are ormaled Pare-wdow dey emae baed o he e of 5 ample 0 8 0

6 Pare Wdow Emao Pare Wdow Emao Pare wdow emae p of a uvarae ormal dey Pare wdow emae p of a bvarae ormal dey o. of ample Wdow wdh h.0 h0.5 h Pare Wdow Emao Choog he Badwdh: Uvarae Cae Pare wdow emae p of a bvarae ormal dey The problem of choog he badwdh crucal dey emao A mall badwdh wll yeld a dey emae ha py ad very hard o erpre A large badwdh wll over-mooh he dey ad ma he rucure he daa 4

7 Choog he Badwdh: Uvarae Cae Choog he Badwdh: Uvarae Cae 5 7 Choog he Badwdh: Uvarae Cae Mulvarae Dey Emao The derved epreo of he emae P for mulple dmeo wa Noce ha he badwdh h he ame for all he ae, o h dey emae wll be wegh all he a equally However, f he pread of he daa much greaer oe of he coordae dreco ha he oher, we hould ue a vecor of moohg parameer or eve a full covarace mar, whch complcae he procedure There are wo bac alerave o olve he calg problem whou havg o ue a more geeral erel dey emae Pre-cale each a ormale o u varace, for ace Pre-whe he daa learly raform o have u covarace mar, emae he dey, ad he raform bac [Fuuga] The wheg raform mply yl -/ M T, where L ad M are he egevalue ad egevecor marce of he ample covarace of Fuuaga mehod equalvale o ug a hyper-ellpodal erel 6 8

8 9 Produc Kerel 30 Produc Kerel, Eample 3 Produc Kerel, Eample 3 Clafcao baed o Pare Wdow Defg a Kerel Pare wdow fuco he Pare wdow dey, p, become he arhmec um of Gaua, Tragular, Recagular fuco are powerfully ued a Kerel Pare wdow fuco Deco fuco for -cla problem h φ δ p δ p p p p p p d d d ˆ ˆ δ ω δ ω ω ω Pare wdow ormaled where,...,,,...,, :, ω ω δ

9 Naïve Baye clafer - Neare Neghbor Clafer - NNC Neare Neghbor dey emao The Neare Neghbor clafcao rule Characerc of he NN clafer Opmg he NN clafer Clafcao baed o Pare Wdow The deco boudare a -D Pare wdow dchoomer deped o he wdow wdh h. No-paramerc Dey Emao: Revew Recall from he prevou lecure ha he geeral epreo for oparamerc dey emao mall h large h / p where he volume urroud g he oal umber of eample he umber of eample de A ha me, we meoed ha h emae could be compued by Fg he volume ad deermg he umber of daa po de Th he approach ued Kerel Dey Emao Pare wdow Fg he value of ad deermg he mmum volume ha ecompae po he daae Th gve re o he Neare Neghbor NN approach A mall h lead o boudare ha are more complcaed ha for large h o ame daa e 34 36

10 -NN Noparamerc Emao I he NN mehod we grow he volume urroudg he emao po ul ecloe a oal of daa po The dey emae he become -NN Noparamerc Emao The -NN emae of -D dey for 5 p / πr / p πr NN Noparamerc Emao -NN Noparamerc Emao Egh po -D & -NN dey emae, for 3 & 5 / p coa Bmodal Drbuo Gaua Drbuo 38 40

11 -NN Dey Emao, Eample -NN Dey Emao, Eample b To llurae he behavor of NN we geeraed everal dey emae for a uvarae mure of wo Gaua: P½ N0,+½ N0,4ad everal value of N ad NN Dey Emao, Eample a The performace of he NN dey emao echque o wo dmeo lluraed hee fgure The op fgure how he rue dey, a mure of wo bvarae Gaua p N µ, Σ + N µ, Σ 0 µ 5 wh 5 µ 0 Σ Σ 4 The boom fgure how he dey emae for 0 eghbor ad N00 eample I he e lde we how he coour of he wo drbuo overlapped wh he rag daa ued o geerae he emae -NN Emao a Bayea Clafer The ma advaage of he NN mehod ha lead o a very mple appromao of he opmal Baye clafer Aume ha we have a daae wh eample, from cla ω, ad ha weare ereed clafyg a uow ample u We draw a hyper-phere of volume aroud u. Aume h volume coa a oal of eample, from cla ω We ca he appromae he lelhood fuco ug he NN mehod by: / p ω Smlarly, he ucodoal dey emaed by / p Ad he pror are appromaed by pω Pug everyhg ogeher, he Baye clafer become p ω p ω p ω p ample cluded ω / / Rego R wh volume capured ample 4 44

12 The Neare Neghbor Clafcao Rule The Neare Neghbor Rule NN a very uve mehod ha clafe ulabeled eample baed o her mlary o eample he rag e For a gve ulabeled eample R d, fd he cloe labeled eample he rag daa e ad ag u o he cla ha appear mo frequely wh he - ube The NN oly requre A eger A e of labeled eample rag daa A merc o meaure cloee The Neare Neghbor RuleNNR Approach The NNR approach lead o a parog he feaure o oroo cell cog of all po cloer o gve rag po ha o ay oher po -Dmeo 3-Dmeo Eample we have hree clae ad he goal o fd a cla label for he uow eample u I h cae we ue he Eucldea dace ad a value of 5 eghbor Of he 5 cloe eghbor, 4 belog o ω ad belog o ω 3, o u aged o ω, he predoma cla The Neare Neghbor RuleNNR Approach -NN Aco: Eample 3-NNC NNC 46 48

13 -NN Aco: Eample NN veru NN NN veru NN The ue of large value of ha wo ma advaage Yeld mooher deco rego Provde probablc formao The rao of eample for each cla gve formao abou he ambguy of he deco However, oo large a value of dermeal I deroy he localy of he emao ce farher eample are ae o accou I addo, creae he compuaoal burde 49 5 Characerc of he NN Clafer NN veru NN Advaage Aalycally racable Smple mplemeao Nearly opmal he large ample lm N 8 P[error] Baye < P[error] NN < P[error] Baye Ue local formao, whch ca yeld hghly adapve behavor Led elf very ealy o parallel mplemeao Dadvaage Large orage requreme Compuaoally eve recall Hghly ucepble o he cure of dmeoaly 50 5

14 Opmg Sorage Requreme Feaure Weghg NN ad he Problem of Feaure Weghg Feaure Weghg 54 56

15 Improvg he Neare Neghbor Search Procedure Dace Clafer Mmum Dace Clafer Mamum Dace Clafer d Tree Eample Merc & Neare-Neghbor Clafcao The eare-eghbor clafer rele o a merc or dace fuco bewee paer A merc D*,* merely a fuco ha gve a geeraled calar dace bewee wo argume paer A merc mu have 4 propere: for all vecor, y, ad. Reflevy D, y 0 f ad oly f y. No egavy D, y 0 3. Symmery D, y D y, 4. Tragle Iequaly D, y D, + D, y 58 60

16 Dace Meaure Oe geeral cla of merc for d -dmeoal paer he Mow merc Compuaoal Compley of he -NNR No-Eucldea dacecy-bloc, Square eaer o compue, bu gve re o ably varao d L, y y Eucldea Eucldea L orm he Mahaa dace L orm he Eucldea dace L 8 orm Recagular dace Square Cy -Bloc 6 63 Dace Meaure Dace fuco eablh a meaure of mlary bewee paer vecor. The d-dmeoal Eucldea dace, L defed by : D, y y y d y Dace Meaure The Tamoo merc fd mo ue aoomy clafcao of pla ad amal ad oology clafcao of deae, where he dace bewee wo e defed a D Tamoo S, S + + Cy-Bloc dace, L Abolue dace, Mahaa dace D, y y d y where, he umber of eleme Se S he umber of eleme Se S he umber ha boh e Recagular dace, L 8 : { d},,..., D, y ma y y 6 64

17 Deg of Neare Neghbor Clafcao Dace Mamao. Load prooype paer : m,,..., c, m,..., N. Read pu paer. 3. Compue dace : Mamao D, Q m{ D, m{ D, : D,,,, D,, KM}, KM} D, D, D,,,..., c, m,..., N m m m{ D,,, KM} 4. Ag : m m ω D, D, for Q m{ m{ coa ma{ m{ + depede,,, +,, KM}, KM}, KM} forall, KM} Mmum Dace Clafer Mmum dace clafer a effecve ool olvg he paer clafcao problem. Mmum dace clafer compue he dace from a paer o he prooype of each cla,,...,m, ad ag he paer o he cla o whch cloe,.e., aged o cla ω f D, <D, for all? Mamum Dace Clafer The Dcrma fuco ca be epreed a d w + θ where, w wegh vecor, θ ba Deco Rule ω f D, m{ D,,, K, M} where, prooype average or cla ceer for cla w. uow paer. d ad d are value of he dcrma fuco for paer clae ad, repecvely. dd -d 0 wll be he equao defg he urface ha eparae cla ad. The ω,,,..., M d > d 66 68

18 Sgle Prooype Eucldea dace D, bewee a arbrary paer ad he -he prooype The Ucrcal Ue of Eucldea Merc The Ucrcal Ue of Eucldea Merc cao addre he problem of ralao varace D, d The Eucldea dace D, 3 much larger ha D, M We ee a dace meaure ha would be eve o ralao, cale ad roao 69 7 Mulple Prooype For mulple prooype, deco rule become f D, ω D, ω ω m{ D, where, repree he -h caegory; m repree he m-h prooype; N repree he o. of prooype of caegory. m m{ D, ω,, m,..., N },..., M }, ω N Repor The deco urface bewee ay wo clae ω ad ω formed by he perpedcular becor of vecor - Prove ha he deco boudary of ad for -clae hyperplae whch he perpedcular becor of he le og ad. where, d-dmeoal vecor perpedcular becor The dcrma fuco m m d ma{, m,..., N,,..., M} Eample cae d 70 7