Nonparametric Techniques

Transcription

1 Noparametrc Techques

2 Noparametrc Techques w/o assumg ay partcular dstrbuto the uderlyg fucto may ot be kow e.g. mult-modal destes too may parameters Estmatg desty dstrbuto drectly Trasform to a lower-dmesoal space where parametrc techques may apply more o ths later o dmeso reducto 2

3 Eample Estmate the populato growth, aual rafall, etc. the US p,yddy s the probablty of ra fall [,+d,y,y+dy] 3

4 Eample cot. A smple parametrc model for p,y probably does ot est I stead partto the area to a lattce At each,y, cout the amout of ra r,y Do that for a whole year Normalze S r,y = 4

5 Desty estmato probablty p probablty P j p d value j value 5

6 From equato Desty estmato j P p d p From observato j P k Hece p k / j k / V 6

7 Comparso I Realty: The umber of trag samples s lmted f V s too small, k becomes erratc What does 0 mea? f V s too large, p s ot represetatve p k / j k / V I theory: If becomes ftely large, k/ approaches the probablty, p = k//v s the oly a space average Hece, V must be allowed to go to zero as goes to fty 7

8 I Theory Theoretcally, we ca use a sequece of samples wth creasg sze for estmato The p p f V 0 2 lm lm k k 3 lm 0 8

9 Two dfferet approaches Costra the rego sze Shrk the rego to mata good localty Parze Wdows Costra the sample sze Elarge the umber of samples to mata good resoluto K -earest-eghbors 9

10 0 Parze Wdows Use a wdowg fucto, e.g. A sequece of regos ca be defed e or otherwse h h h / h V p h k By defto

11 Parze Wdow cot. As creases The wdow becomes arrower by h The wdow becomes taller by /V Samplg wth smaller aperture but hgher focus The same 00 dollars collected from 00 people ad from perso s dfferet per perso d d u du V h

12 p Small : large aperture, smoothed, fuzzy estmate Large : small aperture, sharp, erratc estmate 2

13

14 Fve samples Wdowg fuc: 2D Samplg 4

15 Wdow sze # of samples 5

16

17 Does t work? Work the sese that you f you are able to shrk dow the wdow sze as much as you wat certaly, you must smultaeously crease the umber of samples avalable, the the lmt of the profle should be the correct probablty Ths mples treatg p as a radom varable Ep =p Varp -> 0 7

18 8 Covergece of Mea ] [ ] [ v v v v v v p d p d p h V h V E p E p Wll p goes to p? If goes to fty wll cover all possble summato to tegrato wth p dstrbuto weghted by p Sample v appears wth probablty pv

19 9 Covergece of Varace Wll p always ed up at p for certa? V must approach fty, eve V whe goes to zero V p d p h V V p d p h V V p h V E p h V E sup sup v v v v v v -> 0 as -> fty

20 k -earest-eghbor Parze wdow sze hard to estmate Costra the umber of data tems stead of the sze of the wdow k elarge wdow aroud to eclose that may samples, the p k / V 20

21 k -earest-eghbor Itutvely, as creases k should crease for good represetato V should decrease for good localzato The followg codtos guaratee covergece lm k lm k 0 2

22 Sharp spkes aroud data pots: K=, the probablty estmate s fty at data pot rego sze s zero to capture sample 22

23 23

24 k 256 k 6 6 k 4 k 24

25 25 A Eample Estmatg tagged samples a volume V aroud captures k samples, of them are p k k k V k V k V k V k p p p V k p c j c j j / / / /,, /,

26 Parametrc Comparso smple ad aalytcal may ot ft well real-world destes No-parametrc fleble ad ft all destes eed to remember all samples 26

27 Oe Fal Note Here we talk about Parze wdow ad k - earest-eghbor rule as a way to estmate a sgle probablty desty Ths rule s equally useful at labelg a sample agast multple probable classes destes More o that lear dscrmat fucto 27

28 More Realstc Scearos Drake s Equato Rate of start formato, fracto of stars havg plaets, average # of plaets per star that support lfe, fracto of such stars actually develop lfe, fractos of such stars actually develop cvlzato, such cvlzato have commucato, legth of tme such cvlzato actually release sgals 28

29 More Realstc Scearos Chace of a perso develops cacer acestry, brth place, how rased, lvg habts, educato hstory, work hstory, eercse habt, come, debt, food take, etc. Chace of a perso cotrbutes to poltcal campag 29

30 Curse of Dmesoalty Not possble to estmate dstrbutos such hgh-dmesoal space # of samples eeded are geerally ftely large 30

31 X = rad3,3 Practcal Usage Samplg based o certa dstrbuto default s uform Need to evaluate certa epectato Techology advaces by ale cotact Lfe epectacy for cacer case Amout of moey for poltcal campags 3

32 Geeral Idea Fte umber samples: sample mea/varace to estmate populato mea/varace z l, l =,, L Samples may ot be depedet Some dstrbuto uform s easer to sample tha others fz s small regos where pz s large ad vce versa 32

33 From Oe to Aother z: uform y: ay kow dstrbuto Sample z uformly == Sample y based o py 33

34 Mult-Dmesoal Much more dffcult Do ot kow the form Caot get eough samples to populate the ladscape How to geerate IID samples? 34

35 Rejecto Samplg A real dstrbuto pz A proposal dstrbuto qz Procedure Geerate z o from qz Geerate u o from [0, kqz o ] uformly Reject sample f Otherwse, accept 35

36 Importace Samplg A real dstrbuto pz A proposal dstrbuto qz Procedure Geerate z o from qz, othg rejected pz l /qz l : mportace weght to accout for samplg from wrog dstrbuto 36

37 Image MCMC A very hgh-dmesoal space Samples occupy low-dmesoal mafold such a hgh-dmesoal space Choose a radom start pot Wader about the space, seekg out places wth sample Wth rght seek strategy, samples geerated alog the walk have the rght populato characterstcs 37

38 MCMC Successve samplg pots are NOT depedet, but form a Markov cha Z* s geerated at each step, accepted f probablty > preset threshold Ca be show that the dstrbuto of z t teds to pz as t -> fty So dstrbuto of steps z s after some tal steps ca be used to appromate pz For Metropols algorthm, q has to be symmetrcal qa b=qb a 38

39 Meropols - Hastgs f: proportoal to p target dstrbuto Gve: o : frst sample Q : Markov process to geerate et sample gve curret sample, Q must be symmetrcal e.g., Gaussa Iterato: X pckg from Q r=f /f >= accept, otherwse accept wth prob r. If rejected, = 39

40 Ituto 40

41 Gbbs Samplg Specal case of MCMC Metropols- Hastgs From to + by compoet-wde samplg, j-th varable + depeds o to j- +-th teratos j+ to -th terato 4

42 Slce Samplg Radom walk uder the probablty curve Start from a o wth f>0 Radomly select heght y, 0<y<=f Radomly select le wth the slce, repeat f o slce y o 42