INTRODUCTION TO MACHINE LEARNING 3RD EDITION

Transcription

1 ETHEM ALPAYDIN The MIT Press, 2014 Lecure Slides for INTRODUCTION TO MACHINE LEARNING 3RD EDITION hp:// CHAPTER 2: SUPERVISED LEARNING

2 Learning a Class from Examples 2 Class C of a family car Predicion: Is car x a family car? Knowledge exracion: Wha do people expec from a family car? Oupu: Posiive (+) and negaive ( ) examples Inpu represenaion: x 1 : price, x 2 : engine power

3 Training se X 3 N X { x,r } 1 r 1 if x is posiive 0 if x is negaive x x x 1 2 3

4 Class C 4 p price p AND e engine power e

5 Family Car Decision Tree Yes HP>200 No NO HP>100 Yes No Yes NO Price>24K No YES Yes

6 6 Hypohesis class H H, he hypohesis class (he se of recangles) from which we believe C is drawn 1 if h says x is posiive h( x) 0 if h says x is negaive Error of h on H N E ( h X ) 1 h x r 1 where 1(a b) is 1 if a b and is 0 if a = b empirical error Generalizaion: how well our hypohesis will correcly classify fuure examples ha are no par of he raining se.

7 S, G, and he Version Space 7 The mos specific hypohesis, S, he ighes recangle ha includes all he posiive examples and none of he negaive examples The mos general hypohesis, G h H, beween S and G is consisen and make up he version space (Michell, 1997)

8 Margin 8 Choose h wih larges margin. I seems inuiive o choose h halfway beween S and G; his is o increase he margin, which is he disance beween he margin boundary and he insances closes o i.

9 Doub 9 In some applicaions, a wrong decision may be very cosly and in such a case, we can say ha any insance ha falls in beween S and G is a case of doub, which we canno label wih cerainy due o lack of daa. In such a case, he sysem rejecs he insance and defers he decision o a human exper.

10 Shaering Insances A hypohesis space is said o shaer a se of insances iff for every pariion of he insances ino posiive and negaive, here is a hypohesis ha produces ha pariion. For example, consider 2 insances described using a single real-valued feaure being shaered by inervals. x y + _ x,y x y y x x,y 10

11 Shaering Insances (con) Bu 3 insances canno be shaered by a single inerval. x y z Canno do + _ x,y,z x y,z y x,z x,y z x,y,z y,z x z x,y x,z y Since here are 2 m pariions of m insances, in order for H o shaer insances: H 2 m. 11

12 VC Dimension An unbiased hypohesis space shaers he enire insance space. The larger he subse of X ha can be shaered, he more expressive he hypohesis space is, i.e. he less biased. The Vapnik-Chervonenkis dimension, VC(H) of hypohesis space H defined over insance space X is he size of he larges finie subse of X shaered by H. If arbirarily large finie subses of X can be shaered hen VC(H) = If here exiss a leas one subse of X of size d ha can be shaered hen VC(H) d. If no subse of size d can be shaered, hen VC(H) < d. For a single inervals on he real line, all ses of 2 insances can be shaered, bu no se of 3 insances can, so VC(H) = 2. Since H 2 m, o shaer m insances, VC(H) log 2 H 12

13 VC Dimension N poins can be labeled in 2 N ways as +/ H shaers N if here exiss h ε H consisen for any of hese. Tha is, any learning problem definable by N examples can be learned wih no error by a hypohesis drawn from H. The maximum number of poins ha can be shaered by H is called he Vapnik- Chervonenkis (VC) dimension. An axis-aligned recangle shaers 4 poins only! 13

14 VC Dimension Example Consider axis-parallel recangles in he real-plane, i.e. conjuncions of inervals on wo real-valued feaures. Some 4 insances can be shaered. Some 4 insances canno be shaered: 14

15 VC Dimension Example (con) No five insances can be shaered since here can be a mos 4 disinc exreme poins (min and max on each of he 2 dimensions) and hese 4 canno be included wihou including any possible 5 h poin. Therefore VC(H) = 4 Generalizes o axis-parallel hyper-recangles (conjuncions of inervals in n dimensions): VC(H)=2n. 15

16 Probably Approximaely Correc (PAC) Learning The only reasonable expecaion of a learner is ha wih high probabiliy i learns a close approximaion o he arge concep. In he PAC model, we specify wo small parameers, ε and δ, and require ha wih probabiliy a leas (1 δ) a sysem learn a concep wih error a mos ε. 16

17 Formal Definiion of PAC-Learnable Consider a concep class C defined over an insance space X conaining insances of lengh n, and a learner, L, using a hypohesis space, H. C is said o be PAC-learnable by L using H iff for all cc, disribuions D over X, 0<ε<0.5, 0<δ<0.5; learner L by sampling random examples from disribuion D, will wih probabiliy a leas 1 δ oupu a hypohesis hh such ha error D (h) ε, in ime polynomial in 1/ε, 1/δ, n and size(c). 17

18 Issues of PAC Learnabiliy The compuaional limiaion also imposes a polynomial consrain on he raining se size, since a learner can process a mos polynomial daa in polynomial ime. How o prove PAC learnabiliy: Firs prove sample complexiy of learning C using H is polynomial. Second prove ha he learner can rain on a polynomial-sized daa se in polynomial ime. To be PAC-learnable, here mus be a hypohesis in H wih arbirarily small error for every concep in C, generally C H. 18

19 19 Probably Approximaely Correc (PAC) Learning (2) How many raining examples N should we have, such ha wih probabiliy a leas 1 δ, h has error a mos ε? (Blumer e al., 1989) Each srip is a mos ε/4 Pr ha we miss a srip 1 ε/4 Pr ha N insances miss a srip (1 ε/4) N Pr ha N insances miss 4 srips 4(1 ε/4) N (1 x) exp( x) 4(1 ε/4) N δ 4exp( εn/4) δ and N (4/ε)log(4/δ)

20 N (4/ε)log(4/δ) 20 Therefore, provided ha we ake a leas (4/ε)log(4/δ) independen examples from C and use he ighes recangle as our hypohesis h, wih confidence probabiliy a leas 1 δ, a given poin will be misclassified wih error probabiliy a mos ε. We can have arbirary large confidence by decreasing δ and arbirary small error by decreasing ε, and we see in above equaion ha he number of examples is a slowly growing funcion of 1/ε and 1/δ, linear and logarihmic, respecively.

21 Noise and Model Complexiy 21 Imprecision in recording he inpu aribues Errors in labeling he daa poins May be addiional aribues, which we have no aken ino accoun,

22 Noise and Model Complexiy 22 Use he simpler one because Simpler o use (lower compuaional complexiy) Easier o rain (lower space complexiy) Easier o explain (more inerpreable) Generalizes beer (lower variance - Occam s razor) Noe: A simpler model has more bias. Finding he opimal model corresponds o minimizing boh he bias and he variance. Occam s razor: Simpler explanaions are more plausible and any unnecessary complexiy should be shaved off.

23 Muliple Classes, C i,i=1,...,k X 1 N { x, r }, r is K-dim r i 1 if x Ci 0 if x C j, j i Train hypoheses h i (x), i =1,...,K: h i x 1 if x Ci 0 if x C j, j i The oal empirical error 23

24 Regression 24 In classificaion, given an inpu, he oupu ha is generaed is Boolean; i is a yes/no answer. If he oupu is coninuous and here is no noise he N ask is inerpolaion. X x, r, r, r f x In regression, here is noise added o he oupu of he unknown funcion r f x The explanaion for noise is ha here are exra hidden variables ha we canno observe z denoe hose hidden variables. r f 1 x, z *

25 Regression We would like o approximae he oupu by our model g(x). The empirical error is: N 1 E g r g N X x g x w x w 2 g x w x w x w If g(x) is linear: x 1 1 d d 0 g w x w x w N 1 E w w r w x w N, X Error minimizaion w 1 x r x w r w x xrn Nx 2,

26 Model Selecion & Generalizaion 26 Learning is an ill-posed problem when; daa is no sufficien o find a unique soluion. Exp: There are 2 d possible ways o wrie d binary values and herefore, wih d inpus, he raining se has a mos 2 d examples. Afer seeing N example cases, here remain 22d N possible funcions. The need for inducive bias, assumpions abou H; make some exra assumpions o have a unique soluion wih he daa we have.

27 27 Assuming he shape of a recangle is one inducive bias, and hen he recangle wih he larges margin for example, is anoher inducive bias. Generalizaion: How well a model performs on new daa Overfiing: H more complex han C or f Underfiing: H less complex han C or f

28 Underfiing and Overfiing Underfiing Overfiing Complexiy of a Decision Tree = number of nodes i uses Complexiy of he classificaion funcion Underfiing: when model is oo simple, boh raining and es errors are large Overfiing: when model is oo complex and es errors are large alhough raining errors are small.

29 Triple Trade-Off 29 There is a rade-off beween hree facors (Dieerich, 2003): 1. Complexiy of H, c(h), 2. Training se size, N, 3. Generalizaion error, E, on new daa As N, E As c(h), firs E and hen E

30 30 Cross-Validaion Error on new examples; acually he esing error is used as an esimaion of he generalizaion error! Two errors: raining error, and esing error usually called generalizaion error. Typically, he raining error is smaller han he generalizaion error. To esimae generalizaion error, we need daa unseen during raining. We could spli he daa as Training se (50%) Validaion se (25%)opional, for selecing ML algorihm parameers (e.g. model complexiy) Tes (publicaion) se (25%) Resampling when here is few daa

31 Dimensions of a Supervised Learner 31 X N, 1 x r The sample is independen and idenically disribued (iid); he ordering is no imporan and all insances are drawn from he same join disribuion p(x,r). The aim is o build a good and useful approximaion o r using he model g(x θ). Three following decisions we mus make: 1. Model: g x θ where g( ) is he model, x is he inpu, and θ are he parameers. 31

32 32 The model (inducive bias), or H, is fixed by he machine learning sysem designer based on his or her knowledge of he applicaion and he hypohesis h is chosen (parameers are uned) by a learning algorihm using he raining se, sampled from p(x,r). 2. Loss funcion: The approximaion error, or loss, is he sum of losses over he individual insances X, x E L r g 3. Opimizaion procedure: To find θ ha minimizes he oal error * arg min E X Remark: This procedure is ypical for Parameric approaches o supervised learning; Non-parameric approaches work differenly!

33 Condiions 33 For his seing o work well, he following condiions should be saisfied: The hypohesis class of g( ) should be large enough, ha is, have enough capaciy, o include he unknown funcion ha generaed he daa ha is represened in X in a noisy form. There should be enough raining daa o allow us o pinpoin he correc (or a good enough) hypohesis from he hypohesis class. We should have a good opimizaion mehod ha finds he correc hypohesis given he raining daa.