Regularization. Stephen Scott and Vinod Variyam. Introduction. Outline. Machine. Learning. Problems. Measuring. Performance.

Transcription

1 leaning can geneally be distilled to an optimization poblem Choose a classifie (function, hypothesis) fom a set of functions that minimizes an objective function Clealy we want pat of this function to measue pefomance on the taining set, but this is insufficient Measues / 52 sscott@cse.unl.edu Measues 2 / 52 Types of machine leaning poblems functions pefomance vs taining set pefomance genealization pefomance Supevised : Algoithm is given labeled taining data and is asked to infe a function (hypothesis) fom a family of functions (e.g., set of all As) that is able to pedict well on new, unseen examples Classification: Labels come fom a finite, discete set Regession: Labels ae eal-valued Unsupevised : Algoithm is given data without labels and is asked to model its stuctue Clusteing, density estimation Reinfocement : Algoithm contols an agent that inteacts with its envionment and leans good actions in vaious situations Measues 3 / 52 Measues 4 / 52 Examples of Functions In any leaning poblem, need to be able to quantify pefomance of an algoithm In supevised leaning, we often use a loss function (o eo function) J fo this task Given instance x with tue label y, if the leane s pediction on x is ŷ, then is the loss on that instance J (y, ŷ) - : J (y, ŷ) = if y 6= ŷ, othewise Squae : J (y, ŷ) =(y ŷ) 2 Coss-Entopy: J (y, ŷ) = y ln ŷ ( y) ln ( ŷ) (y and ŷ ae consideed pobabilities of a label; genealizes to multi-class.) Hinge : J (y, ŷ) =max(, y ŷ) (used sometimes fo lage magin classifies like SVMs) All non-negative 5 / 52 6 / 52

2 Taining Expected Given a loss function J and a taining set X, the total loss of the classifie h on X is eo X (h) = X x2x J (y x, ŷ x ), whee y x is x s label and ŷ x is h s pediction Moe impotantly, the leane needs to genealize well: Given a new example dawn iid accoding to unknown pobability distibution D, we want to minimize h s expected loss: eo D (h) =E x D [J (y x, ŷ x )] Is minimizing taining loss the same as minimizing expected loss? 7 / 52 8 / 52 Expected vs Taining x 2 Sufficiently sophisticated leanes (decision tees, multi-laye As) can often achieve abitaily small (o zeo) loss on a taining set A hypothesis (e.g., A with specific paametes) h ovefits the taining data X if thee is an altenative hypothesis h such that eo X (h) < eo X (h ) h 2 h x and eo D (h) > eo D (h ) 9 / 52 / 52 / 52 y: pice x: milage To genealize well, need to balance taining accuacy with simplicity Multitask s 2 / 52 Geneally, if the set of functions H the leane has to choose fom is complex elative to what is equied fo coectly pedicting the labels of X, thee s a lage chance of ovefitting due to the lage numbe of wong choices in H Could be due to an ovely sophisticated set of functions E.g., can fit any set of n eal-valued points with an (n )-degee polynomial, but pehaps only degee 2 is needed E.g., using an A with 5 hidden layes to solve the logical AD poblem Could be due to taining an A too long Ove-taining an A often leads to weights deviating fa fom zeo Makes the function moe non-linea, and moe complex Often, a lage data set mitigates the poblem

3 : Ovetaining Multitask s 3 / 52 Eo Eo vesus weight updates (example ) Taining set eo Validation set eo umbe of weight updates Multitask s 4 / 52 Eo Eo Eo vesus weight updates (example ) Taining set eo Validation set eo umbe of weight updates Eo vesus weight updates (example 2) Taining set eo Validation set eo umbe of weight updates Dange of stopping too soon Patience paamete in Algoithm 7. Can e-stat with small, andom weights (Algoithm 7.2) : L 2 om Still want to minimize taining loss, but balance it against a complexity penalty on the paametes used: J ( ; X, y) =J ( ; X, y)+ ( ) ( ) =(/2)k k 2 2, i.e., sum of squaes of netwok s weights Since = w, this becomes J (w; X, y) =( /2)w > w + J (w; X, y) 2 [, ) weights loss J against penalty As weights deviate fom zeo, activation functions become moe nonlinea, which is highe isk of ovefitting Multitask Multitask s s 5 / 52 6 / 52 Measues Measues : L 2 om : L 2 om Related to ealy stopping: Fo linea model and squae loss, get tajectoy of weight updates that, on aveage, lands in a simila esult Multitask s 7 / 52 w is optimal fo J, optimal fo egulaize J less sensitive to w, so w (optimal fo J ) close to than w 2 Multitask s 8 / 52 Length of tajectoy elated to weight in L 2 egulaize Measues Measues

4 : L om Multitask s 9 / 52 ( ) =k k, i.e., sum of absolute values of netwok s weights J (w; X, y) = kwk + J (w; X, y) As with L 2 egulaization, penalizes lage weights Unlike L 2 egulaization, can dive some weights to zeo Spase solution Sometimes used in featue selection (e.g., LASSO algoithm) Multitask s 2 / 52 If H is poweful and X is small, then a leane can choose some h 2 H that fits the idiosyncasies o noise in the data Deep As would like to have at least thousands o tens of thousands of data points In classification of high-dimensional data (e.g., image classification), expect the classifie to toleate tansfomations and noise ) Can atificially enlage data set by duplicating existing instances and applying tansfomations Tanslating, otating, scaling Be caeful to to change the class, e.g., b vs d o 6 vs 9 ) Can also apply noise injection to input o even hidden layes Multitask Multitask s 2 / 52 If multiple tasks shae geneic paametes, initially pocess inputs via shaed nodes, then do final pocessing via task-specific nodes Backpopagation woks as befoe with multiple output nodes Seves as a egulaize since paamete tuning of shaed nodes is based on backpopagated eo fom multiple tasks Multitask s 22 / 52 Imagine if, fo a netwok, we could aveage ove all netwoks with each subset of nodes deleted Analogous to bagging, whee we aveage ove As tained on andom samples of X In each taining iteation, sample a andom bit vecto µ, which detemines which nodes ae used (e.g., P(µ i = ) =.8 fo input unit,.5 fo hidden unit) Make pedictions by sampling new vectos µ and aveaging Appoaches Appoaches (cont d) Multitask s Paamete Tying: If two leanes ae leaning the same task but diffeent distibutions, can tie thei paametes togethe If w (A) ae weights fo task A and w (B) ae weights fo task B, then can use egulaization tem (w (A), w (B) )=kw (A) w (B) k 2 2 Paamete Shaing: When detecting objects in an image, the same ecognize should apply invaiant to tanslation Can tain a single detecto (subnetwok) fo the object (e.g., cat) by taining full netwok on multiple images with tanslated cats, whee the cat detecto subnetwoks shae paametes (single copy, used multiple times) Multitask s Spase Repesentations: Instead of penalizing lage weights, penalize lage outputs of hidden nodes: J ( ; X, y) =J ( ; X, y)+ (h), whee h is the vecto of hidden unit outputs 23 / / 52

5 Befoe setting up an expeiment, need to undestand exactly what the goal is Estimate the genealization pefomance of a hypothesis Tuning a leaning algoithm s paametes two leaning algoithms on a specific task Etc. Will neve be able to answe the question with % cetainty Due to vaiances in taining set selection, test set selection, etc. Will choose an estimato fo the quantity in question, detemine the pobability distibution of the estimato, and bound the pobability that the estimato is way off Estimato needs to wok egadless of distibution of taining/testing data eed to note that, in addition to statistical vaiations, what we detemine is limited to the application that we ae studying E.g., if A bette than A 2 on speech ecognition, that means nothing about video analysis In planning expeiments, need to ensue that taining data not used fo evaluation I.e., don t test on the taining set! Will bias the pefomance estimato Also holds fo validation set used fo ealy stopping, tuning paametes, etc. Validation set seves as pat of taining set, but not used fo model building 25 / / 52 (cont d) Let eo D (h) be - loss of h on instances dawn accoding to distibution D. If V contains examples, dawn independently of h and each othe 3 Let eo D (h) be - loss of h on instances dawn accoding to distibution D. If V contains examples, dawn independently of h and each othe 3 Then with appoximately 95% pobability, eo D (h) lies in Then with appoximately c% pobability, eo D (h) lies in 27 / 52 eov (h)( eo V (h)) eo V (h) ±.96 E.g. hypothesis h misclassifies 2 of the 4 examples in test set V: eo V (h) = 2 4 =.3 Then with appox. 95% confidence, eo D (h) 2 [.58,.442] 28 / 52 Why? eov (h)( eo V (h)) eo V (h) ± z c %: 5% 68% 8% 9% 95% 98% 99% z c : eo V (h) is a Random Vaiable Binomial Pobability Distibution 29 / 52 Repeatedly un the expeiment, each with diffeent andomly dawn V (each of size ) Pobability of obseving misclassified examples: P() P() = Binomial distibution fo n = 4, p = eo D (h) ( eo D (h)) I.e., let eo D (h) be pobability of heads in biased coin, then P() =pob. of getting heads out of flips 3 / 52 P() = p ( p)! =!( )! p ( p) Pobability P() of heads in coin flips, if p = P(heads) Expected, o mean value of X, E[X] (= # heads on flips = # mistakes on test exs), is X E[X] ip(i) =p = eo D (h) Vaiance of X is i= Va(X) E[(X E[X]) 2 ]=p( p) Standad deviation of X, X, is q E[(X E[X]) 2 ]= p p( p) X

6 Appoximate Binomial Dist. with omal omal Pobability Distibution 3 / 52 eo V (h) =/ is binomially distibuted, with mean µ eov (h) = eo D (h) (i.e., unbiased est.) standad deviation eov (h) eod (h)( eo D (h)) eo V (h) = (inceasing deceases vaiance) Want to compute confidence inteval = inteval centeed at eo D (h) containing c% of the weight unde the distibution Appoximate binomial by nomal (Gaussian) dist: mean µ eov (h) = eo D (h) standad deviation eov (h) eov (h)( eo V (h)) eo V (h) 32 / omal distibution with mean, standad deviation ! p(x) = p exp x µ The pobability that X will fall into the inteval (a, b) is given by R b a p(x) dx Expected, o mean value of X, E[X], is E[X] =µ Vaiance is Va(X) = 2, standad deviation is X = omal Pobability Distibution (cont d) omal Pobability Distibution (cont d) % of aea (pobability) lies in µ ±.28 c% of aea (pobability) lies in µ ± z c c%: 5% 68% 8% 9% 95% 98% 99% z c : Can also have one-sided bounds: c% of aea lies <µ+ z c o >µ z c, whee z c = z ( c)/2 c%: 5% 68% 8% 9% 95% 98% 99% z c: / / 52 Revisited Cental Limit Theoem 35 / 52 If V contains 3 examples, indep. of h and each othe Then with appoximately 95% pobability, eo V (h) lies in eod (h)( eo D (h)) eo D (h) ±.96 Equivalently, eo D (h) lies in eod (h)( eo D (h)) eo V (h) ±.96 which is appoximately eov (h)( eo V (h)) eo V (h) ±.96 (One-sided bounds yield uppe o lowe eo bounds) 36 / 52 How can we justify appoximation? Conside set of iid andom vaiables Y,...,Y, all fom abitay pobability distibution with mean µ and finite vaiance 2. Define sample mean Ȳ (/) P n i= Y i Ȳ is itself a andom vaiable, i.e., esult of an expeiment (e.g., eo S (h) =/) Cental Limit Theoem: As!, the distibution govening Ȳ appoaches nomal distibution with mean µ and vaiance 2 / Thus the distibution of eo S (h) is appoximately nomal fo lage, and its expected value is eo D (h) (Rule of thumb: 3 when estimato s distibution is binomial; might need to be lage fo othe distibutions)

7 Calculating 37 / 52 Pick paamete to estimate: eo D (h) (- loss on distibution D) 2 Choose an estimato: eo V (h) (- loss on independent test set V) 3 Detemine pobability distibution that govens estimato: eo V (h) govened by binomial distibution, appoximated by nomal when 3 4 Find inteval (L, U) such that c% of pobability mass falls in the inteval Could have L = o U = Use table of z c o z c values (if distibution nomal) Distibution 38 / 52 What if we want to compae two leaning algoithms L and L 2 (e.g., two A achitectues, two egulaizes, etc.) on a specific application? Insufficient to simply compae eo ates on a single test set Use K-fold coss validation and a paied t test K-Fold Coss Validation K-Fold Coss Validation (cont d) Distibution 39 / 52 Patition data set X into K equal-sized subsets X, X 2,...,X K, whee X i 3 2 Fo i fom to K, do (Use X i fo testing, and est fo taining) V i = X i 2 T i = X\X i 3 Tain leaning algoithm L on T i to get h i 4 Tain leaning algoithm L 2 on T i to get h 2 i 5 Let p j i be eo of hj i on test set V i 6 p i = p i p 2 i 3 Eo diffeence estimate p =(/K) P K i p i Distibution 4 / 52 ow estimate confidence that tue expected eo diffeence < ) Confidence that L is bette than L 2 on leaning task Use one-sided test, with confidence deived fom student s t distibution with K degees of feedom With appoximately c% pobability, tue diffeence of expected eo between L and L 2 is at most whee p + t c,k s p v u s p t KX (p i p) 2 K(K ) i= Distibution (One-Sided Test) Caveat Distibution 4 / 52 If p + t c,k s p < ou assetion that L has less eo than L 2 is suppoted with confidence c So if K-fold CV used, compute p, look up t c,k and check if p < t c,k s p One-sided test; says nothing about L 2 ove L Distibution 42 / 52 Say you want to show that leaning algoithm L pefoms bette than algoithms L 2, L 3, L 4, L 5 If you use K-fold CV to show supeio pefomance of L ove each of L 2,...,L 5 at 95% confidence, thee s a 5% chance each one is wong ) Thee s an ove 8.5% chance that at least one is wong ) Ou oveall confidence is only just ove 8% eed to account fo this, o use moe appopiate test

8 Moe Specific Measues Confusion Matices So fa, we ve looked at a single eo ate to compae hypotheses/leaning algoithms/etc. This may not tell the whole stoy: test examples: 2 positive, 98 negative h gets 2/2 pos coect, 965/98 neg coect, fo accuacy of ( )/(2 + 98) =.967 Petty impessive, except that always pedicting negative yields accuacy =.98 Would we athe have h 2, which gets 9/2 pos coect and 93/98 neg, fo accuacy =.949? Depends on how impotant the positives ae, i.e., fequency in pactice and/o cost (e.g., cance diagnosis) Beak down eo into type: tue positive, etc. Pedicted Class Tue Class Positive egative Total Positive tp : tue positive fn : false negative p egative fp : false positive tn : tue negative n Total p n Genealizes to multiple classes Allows one to quickly assess which classes ae missed the most, and into what othe class Measues 43 / 52 Measues 44 / 52 ROC Cuves ROC Cuves Plotting tp vesus fp Conside classification via A + linea theshold unit omally theshold f (x; w, b) at, but what if we changed it? Keeping w fixed while changing theshold = fixing hypeplane s slope while moving along its nomal vecto Conside the always hyp. What is fp? What is tp? What about the always + hyp? In between the extemes, we plot TP vesus FP by soting the test examples by the confidence values Measues 45 / 52 b ped all! ped all + Get a set of classifies, one pe labeling of test set Simila situation with any classifie with confidence value, e.g., pobability-based Measues 46 / 52 Ex Confidence label Ex Confidence label x x x x x x x x x x ROC Cuves Plotting tp vesus fp (cont d) ROC Cuves Convex Hull TP x5 x TP ID3 naive Bayes FP Measues 47 / 52 x FP Measues 48 / 52 The convex hull of the ROC cuve yields a collection of classifies, each optimal unde diffeent conditions If FP cost = F cost, then daw a line with slope / P at (, ) and dag it towads convex hull until you touch it; that s you opeating point Can use as a classifie any pat of the hull since can andomly select between two classifies

9 ROC Cuves Convex Hull ROC Cuves Miscellany Measues 49 / 52 TP ID3 naive Bayes Can also compae cuves against single-point classifies when no cuves In plot, ID3 bette than ou SVM iff negatives scace; nb neve bette FP Measues 5 / 52 What is the wost possible ROC cuve? One metic fo measuing a cuve s goodness: aea unde cuve (AUC): P x +2P P x 2 I(h(x +) > h(x )) P i.e., ank all examples by confidence in + pediction, count the numbe of times a positively-labeled example (fom P) is anked above a negatively-labeled one (fom ), then nomalize What is the best value? Distibution appoximately nomal if P, >, so can find confidence intevals Catching on as a bette scala measue of pefomance than eo ate ROC analysis possible (though ticky) with multi-class poblems Pecision-Recall Cuves Pecision-Recall Cuves (cont d) Conside infomation etieval task, e.g., web seach As with ROC, can vay theshold to tade off pecision against ecall Measues 5 / 52 pecision = tp/p = faction of etieved that ae positive ecall = tp/p = faction of positives etieved Measues 52 / 52 Can compae cuves based on containment Use F -measue to combine at a specific point, whee weights pecision vs ecall: F ( + 2 pecision ecall ) ( 2 pecision)+ecall