Instance-Based Learning (a.k.a. memory-based learning) Part I: Nearest Neighbor Classification

Transcription

1 Instance-Based earnng (a.k.a. memory-based learnng) Part I: Nearest Neghbor Classfcaton Note to other teachers and users of these sldes. Andrew would be delghted f you found ths source materal useful n gvng your own lectures. Feel free to use these sldes verbatm, or to modfy them to ft your own needs. PowerPont orgnals are avalable. If you make use of a sgnfcant porton of these sldes n your own lecture, please nclude ths message, or the followng lnk to the source repostory of Andrew s tutorals: Comments and correctons gratefully receved. Ronald J. Wllams CSG0 Sprng 007 Contanng selected sldes adapted from the Andrew Moore tutoral wth the same man ttle Orgnals 001, Andrew W. Moore, Modfcatons 003, Ronald J. Wllams 1-Nearest-Neghbor Classfcaton Gven: dataponts (x 1,y 1 ) (x,y )..(x N,y N ), where each x s an attrbute vector and y s the correspondng class label, determned accordng to y =f(x ) for some unknown functon f (possbly corrupted by nose). Gven query pont x q, your job s to predct y = f ( x ). Nearest Neghbor Classfer: Fnd the closest x n the set of dataponts ( nn) = arg mn ( x, x ) q where d s some dstance metrc. Then classfy x q usng y = y ( nn) d q Orgnals 001, Andrew W. Moore, Modfcatons 003, Ronald J. Wllams Instance-based learnng I: Slde 1

2 1-Nearest-Neghbor s an example of. Instance-based learnng A classfer and functon approxmator that has been around snce about To make a predcton, search database for smlar dataponts, and predct based on these nearby ponts. x 1 y 1 x y x 3 y 3.. x n y n Four thngs make a memory-based learner: A dstance metrc How many nearby neghbors to look at? A weghtng functon (optonal) How to ft wth the local ponts? Orgnals 001, Andrew W. Moore, Modfcatons 003, Ronald J. Wllams Instance-based learnng I: Slde 3 Nearest Neghbor Four thngs make a memory based learner: 1. A dstance metrc Eucldan or related generalzatons. How many nearby neghbors to look at? One 3. A weghtng functon (optonal) N/A 4. How to ft wth the local ponts? Just predct the same output as the nearest neghbor Orgnals 001, Andrew W. Moore, Modfcatons 003, Ronald J. Wllams Instance-based learnng I: Slde 4

3 Multvarate Dstance Metrcs Suppose the nput vectors x1, x, xn are two dmensonal: x 1 = ( x 11, x 1 ), x = ( x 1, x ), x N = ( x N1, x N ). One can draw the nearest-neghbor regons n nput space. d (x,x j ) = (x 1 x j1 ) + (x x j ) d (x,x j ) =(x 1 x j1 ) +(3x 3x j ) The relatve scalngs n the dstance metrc affect regon shapes. Orgnals 001, Andrew W. Moore, Modfcatons 003, Ronald J. Wllams Instance-based learnng I: Slde 5 Effect of Axs Scalng Whch data pont s closer to the query pont? x q x q Horzontal axs rescaled by a factor of Orgnals 001, Andrew W. Moore, Modfcatons 003, Ronald J. Wllams Instance-based learnng I: Slde 6 3

4 Eucldean Dstance Metrc Or equvalently, where d(x, x') = d(x, x') = σ1 = 0 0 ( x x' ) 0 0 σ N Other Metrcs Mahalanobs, Rank-based, Correlaton-based (Stanfll+Waltz, Maes Rngo system ) Orgnals 001, Andrew W. Moore, Modfcatons 003, Ronald J. Wllams Instance-based learnng I: Slde 7 0 σ 0 σ (x - x') T (x - x') Notable Dstance Metrcs Mahalanobs (here, Σ on the prevous slde s not necessarly dagonal, but s symmetrc Scaled Eucldan ( ) 1 norm (absolute) Orgnals 001, Andrew W. Moore, Modfcatons 003, Ronald J. Wllams Instance-based learnng I: Slde 8 nfnty (max) norm 4

5 Dstance Metrc: Thngs to Worry about In practce, should at least rescale every axs to have approxmately the same range Gves every feature more nearly equal weght Can use cross-valdaton to fne-tune axs weghtngs or other metrc parameters But try to elmnate rrelevant features Many rrelevant features could domnate the dstance measure and create a bad classfer Cross-valdaton can be very helpful wth ths Orgnals 001, Andrew W. Moore, Modfcatons 003, Ronald J. Wllams Instance-based learnng I: Slde 9 What about nose? If each data pont has a nosy class label, what would be better than usng only the sngle nearest neghbor? Orgnals 001, Andrew W. Moore, Modfcatons 003, Ronald J. Wllams Instance-based learnng I: Slde 10 5

6 k-nearest Neghbor Four thngs make a memory based learner: 1. A dstance metrc Eucldan or related generalzatons. How many nearby neghbors to look at? k 3. A weghtng functon (optonal) N/A 4. How to ft wth the local ponts? Just take the majorty vote among the k nearest neghbors Orgnals 001, Andrew W. Moore, Modfcatons 003, Ronald J. Wllams Instance-based learnng I: Slde 11 Weghted Nearest Neghbor Classfer Four thngs make a memory based learner: 1. A dstance metrc Eucldan. How many nearby neghbors to look at? Potentally all of them 3. A weghtng functon (optonal) Nearby ponts to the query are weghted strongly, far ponts weakly. 4. How to ft wth the local ponts? Gve each pont a vote based on ths weghtng functon. Classfy accordng to ths vote. Orgnals 001, Andrew W. Moore, Modfcatons 003, Ronald J. Wllams Instance-based learnng I: Slde 1 6

7 Weghtng functons et d=d(x,x query ) Then here are some commonly used weghtng functons Orgnals 001, Andrew W. Moore, Modfcatons 003, Ronald J. Wllams Instance-based learnng I: Slde 13 Categorcal Attrbutes If attrbute x[j] can take on one of several dscrete values v 1, v,, v k (and there s no reason to consder some pars of these values as beng closer than some other pars), then a sensble nterpretaton of x [j]-x q [j] n the dstance computaton s 0 f x [j] = x q [j] or 1 f x [j] x q [j] Ths has a smlar effect as when a 1-out-of-k encodng s used for nput to a neural network Orgnals 001, Andrew W. Moore, Modfcatons 003, Ronald J. Wllams Instance-based learnng I: Slde 14 7