ecture Sldes for ETHEM APAYDI The MIT Press, 00 alpaydn@boun.edu.tr http://www.cpe.boun.edu.tr/~ethe/le
Introducton Questons: Assessent of the expected error of a learnng algorth: Is the error rate of - less than %? Coparng the expected errors of two algorths: Is k- ore accurate than MP? Tranng/valdaton/test sets Resaplng ethods: -fold cross-valdaton ecture otes for E Alpaydın 00 Introducton to Machne earnng e The MIT Press (V.0) 3
Algorth Preference Crtera (Applcaton-dependent): Msclassfcaton error, or rsk (loss functons) Tranng te/space coplexty Testng te/space coplexty Interpretablty Easy prograablty Cost-senstve learnng ecture otes for E Alpaydın 00 Introducton to Machne earnng e The MIT Press (V.0) 4
Factors and Response ecture otes for E Alpaydın 00 Introducton to Machne earnng e The MIT Press (V.0) 5
Strateges of Experentaton Response surface desgn for approxatng and axzng the response functon n ters of the controllable factors ecture otes for E Alpaydın 00 Introducton to Machne earnng e The MIT Press (V.0) 6
Gudelnes for M experents A. A of the study B. Selecton of the response varable C. Choce of factors and levels D. Choce of experental desgn E. Perforng the experent F. Statstcal Analyss of the Data G. Conclusons and Recoendatons ecture otes for E Alpaydın 00 Introducton to Machne earnng e The MIT Press (V.0) 7
The need for ultple tranng/valdaton sets {,V } : Tranng/valdaton sets of fold -fold cross-valdaton: Dvde nto k,,=,..., T share - parts Resaplng and -Fold Cross-Valdaton 3 3 T V T V T V 8 ecture otes for E Alpaydın 00 Introducton to Machne earnng e The MIT Press (V.0)
5 Cross-Valdaton 5 0 5 0 5 9 5 9 4 4 3 3 V T V T V T V T V T V T 9 5 tes fold cross-valdaton (Detterch, 998) ecture otes for E Alpaydın 00 Introducton to Machne earnng e The MIT Press (V.0)
Bootstrappng Draw nstances fro a dataset wth replaceent Prob that we do not pck an nstance after draws e 0. 368 that s, only 36.8% s new! ecture otes for E Alpaydın 00 Introducton to Machne earnng e The MIT Press (V.0) 0
Measurng Error Error rate = # of errors / # of nstances = (F+FP) / Recall = # of found postves / # of postves = TP / (TP+F) = senstvty = ht rate Precson = # of found postves / # of found = TP / (TP+FP) Specfcty = T / (T+FP) False alar rate = FP / (FP+T) = - Specfcty ecture otes for E Alpaydın 00 Introducton to Machne earnng e The MIT Press (V.0)
ROC Curve ecture otes for E Alpaydın 00 Introducton to Machne earnng e The MIT Press (V.0)
ecture otes for E Alpaydın 00 Introducton to Machne earnng e The MIT Press (V.0) 3
Precson and Recall ecture otes for E Alpaydın 00 Introducton to Machne earnng e The MIT Press (V.0) 4
Interval Estaton 0 95 96 96 0 95 96 96 z z P P P / /...... ~ Z 5 = { x t } t where x t ~ ( μ, σ ) ~ ( μ, σ /) 00(- α) percent confdence nterval ecture otes for E Alpaydın 00 Introducton to Machne earnng e The MIT Press (V.0)
6 When σ s not known: 0 95 64 0 95 64 z P P P.... S t S t P t S x S t t, /, / ~ / ecture otes for E Alpaydın 00 Introducton to Machne earnng e The MIT Press (V.0)
Hypothess Testng Reect a null hypothess f not supported by the saple wth enough confdence = { x t } t where x t ~ ( μ, σ ) H 0 : μ = μ 0 vs. H : μ μ 0 Accept H 0 wth level of sgnfcance α f μ 0 s n the 00(- α) confdence nterval Two-sded test 0 z z /, / ecture otes for E Alpaydın 00 Introducton to Machne earnng e The MIT Press (V.0) 7
One-sded test: H 0 : μ μ 0 vs. H : μ > μ 0 Accept f Varance unknown: Use t, nstead of z Accept H 0 : μ = μ 0 f 0, z 0 S t t /,, /, ecture otes for E Alpaydın 00 Introducton to Machne earnng e The MIT Press (V.0) 8
Assessng Error: H 0 : p p 0 vs. H : p > p 0 Sngle tranng/valdaton set: Bnoal Test If error prob s p 0, prob that there are e errors or less n valdaton trals s e P e p0 p0 Accept f ths prob s less than - α - α =00, e=0 ecture otes for E Alpaydın 00 Introducton to Machne earnng e The MIT Press (V.0) 9
oral Approxaton to the Bnoal uber of errors s approx wth ean p 0 and var p 0 (-p 0 ) p p 0 0 p 0 ~ Z Accept f ths prob for = e s less than z -α - α ecture otes for E Alpaydın 00 Introducton to Machne earnng e The MIT Press (V.0) 0
t Test Multple tranng/valdaton sets x t = f nstance t sclassfed on fold t Error rate of fold : x p t Wth and s average and var of p, we accept p 0 or less error f p0 ~ t S s less than t α,- ecture otes for E Alpaydın 00 Introducton to Machne earnng e The MIT Press (V.0)
Coparng Classfers: H 0 : μ 0 = μ vs. H : μ 0 μ Sngle tranng/valdaton set: Mcear s Test Under H 0, we expect e 0 = e 0 =(e 0 + e 0 )/ Accept f < α, e0 e0 e 0 e 0 ~ ecture otes for E Alpaydın 00 Introducton to Machne earnng e The MIT Press (V.0)
-Fold CV Pared t Test 0 0 0 0 0 t t t s s p s p H H, /, /, ~ : : n f Accept vs. 3 Use -fold cv to get tranng/valdaton folds p, p : Errors of classfers and on fold p = p p : Pared dfference on fold The null hypothess s whether p has ean 0 ecture otes for E Alpaydın 00 Introducton to Machne earnng e The MIT Press (V.0)
5 cv Pared t Test Use 5 cv to get folds of 5 tra/val replcatons (Detterch, 998) p () : dfference btw errors of and on fold =, of replcaton =,...,5 p p p / s p p 5 s p p / 5 ~ t Two-sded test: Accept H 0 : μ 0 = μ f n (-t α/,5,t α/,5 ) One-sded test: Accept H 0 : μ 0 μ f < t α,5 5 p ecture otes for E Alpaydın 00 Introducton to Machne earnng e The MIT Press (V.0) 4
5 cv Pared F Test 5 5 s p ~ F 0, 5 Two-sded test: Accept H 0 : μ 0 = μ f < F α,0,5 ecture otes for E Alpaydın 00 Introducton to Machne earnng e The MIT Press (V.0) 5
Coparng > Algorths: Analyss of Varance (Anova) H : 0 Errors of algorths on folds,,,...,, ~,..., We construct two estators to σ. One s vald f H 0 s true, the other s always vald. We reect H 0 f the two estators dsagree. ecture otes for E Alpaydın 00 Introducton to Machne earnng e The MIT Press (V.0) 6
7 0 0 SSb H SSb S S H ~ ~ / ˆ /, ~ : we have s true, So when, naely, s Thus anestatorof s true If ecture otes for E Alpaydın 00 Introducton to Machne earnng e The MIT Press (V.0)
8 0 0 F H F SSw SSb SSw SSb SSw S SSw S S S H,,, : ~ / / / / / ~ ~ ˆ f : group varances average of s the secondestator to our Regardlessof ecture otes for E Alpaydın 00 Introducton to Machne earnng e The MIT Press (V.0)
AOVA table 9 If AOVA reects, we do parwse posthoc tests ) ( ~ : vs : 0 w t t H H ecture otes for E Alpaydın 00 Introducton to Machne earnng e The MIT Press (V.0)
Coparson over Multple Datasets Coparng two algorths: Sgn test: Count how any tes A beats B over datasets, and check f ths could have been by chance f A and B dd have the sae error rate Coparng ultple algorths ruskal-walls test: Calculate the average rank of all algorths on datasets, and check f these could have been by chance f they all had equal error If W reects, we do parwse posthoc tests to fnd whch ones have sgnfcant rank dfference ecture otes for E Alpaydın 00 Introducton to Machne earnng e The MIT Press (V.0) 30