CptS 570 Machine Learning School of EECS Washington State University. CptS Machine Learning 1

Transcription

1 CpS 570 Machine Learning School of EECS Washingon Sae Universiy CpS Machine Learning 1

2 Form of underlying disribuions unknown Bu sill wan o perform classificaion and regression Semi-parameric esimaion Disjuncion of local parameric mehods Clusering k-means clusering Non-parameric esimaion Insance-based (lazy) learning Neares-neighbor classifier CpS Machine Learning 2

3 Unsupervised learning Pariion insances ino k disjoin ses Each se has a represenaive insance m i Place insance ino se i such ha disance(, m i ) is minimal Choose new cenral m i for each se Repea unil m i converge CpS Machine Learning 3

4 OR For i = 1 o k C i = { X b i = 1} Find m i C i minimizing Σ( C i ) disance(m i,) 4

5 5

6 Pre-processor for supervised learning Cluser insances ino k ses Learn local classifier for each se New insance classified based on closes se New insance classified by disance-weighed voe over all ses Choosing k? Try several (2 k N) Choose k minimizing inra-cluser disance and maimizing iner-cluser disance For classificaion, use cross-validaion o choose k CpS Machine Learning 6

7 Given raining se X={ } drawn iid from p() Divide daa ino bins of size h Hisogram: pˆ # in he same bin as Nh Naive esimaor: pˆ # h / 2 Nh h / 2 pˆ 1 Nh N 1 w h w u 1 0 if u 1/ 2 oherwise CpS Machine Learning 7

8 8

9 9

10 Kernel funcion, e.g., Gaussian kernel: K u 1 2 ep 2 u 2 Kernel esimaor (Parzen windows) pˆ 1 Nh N 1 K h 10

11 11

12 Insead of fiing bin widh h and couning he number of insances, fi he insances (neighbors) k and check bin widh pˆ k 2Nd d k (), disance o kh closes insance o Like naïve esimaor wih h = 2d k () k 12

13 13

14 Esimae p( C i ) and use Bayes rule Kernel esimaor k-nn esimaor i N d i i i i i i N d i i r h K Nh C P C p g N N C P r h K N h C p ˆ ˆ ˆ ˆ 14 k k p C P C p C P i i i i ˆ ˆ ˆ ˆ k i = # of k neares neighbors of class C i

15 Insances map o poins in R d Less han 20 feaures per insance (d < 20) Los of raining daa Advanages Training is very fas Learn comple arge funcions Don lose informaion Disadvanages Slow a query ime Easily fooled by irrelevan feaures CpS Machine Learning 15

16 CpS Machine Learning 16

17 Euclidean disance for numeric feaures Normalize feaure values Hamming disance for discree feaures Disance = 1 if feaure values differ, else 0 CpS Machine Learning 17

18 Imagine insances described by 20 feaures, bu only 2 are relevan o arge funcion Curse of dimensionaliy: Neares neighbor easily mislead when high-dimensional X One approach Scale j h ais by weigh z j, where z 1,,z d chosen o minimize predicion error Use cross-validaion o auomaically choose weighs Noe: seing z j o zero eliminaes his dimension alogeher CpS Machine Learning 18

19 Time/space compleiy of k-nn is O(N) Find a subse Z of X ha is small and is accurae in classifying X CpS Machine Learning 19

20 Incremenal algorihm: Add insance if needed 20

21 As number of raining eamples approaches infiniy error(k-nn) < 2 * Bayes opimal error As number of raining eamples approaches infiniy and k ges large error(k-nn) = Bayes opimal error CpS Machine Learning 21

22 Aka smoohing models Regressogram gˆ where b, N 1 N b 1, b 1 if, r 0 oherwise is in he same bin wih 22

23 23

24 24

25 When k or h is small, single insances maer Bias is small, variance is large (undersmoohing) High compleiy As k or h increases, we average over more insances Variance decreases bu bias increases (oversmoohing) Low compleiy Cross-validaion is used o fine-une k or h. 25

26 Form of underlying disribuions unknown Base on disance o raining eamples Unsupervised clusering k-means clusering Densiy esimaion Classificaion k-neares neighbor Regression CpS Machine Learning 26