UVA$CS$4501$+$001$/$6501$ $007$ Introduc8on$to$Machine$Learning$ and$data$mining$$ $ Lecture$19:!Decision!Tree!/! Random!Forest!/!
|
|
|
- Silas Blake
- 6 years ago
- Views:
Transcription
1 YanjunQi/UVACS4501L01L6501L07 UVA$CS$4501$+$001$/$6501$ $007$ Introduc8on$to$Machine$Learning$ and$data$mining$$ $ Lecture$19:DecisionTree/ RandomForest/Ensemble$ YanjunQi/Jane,,PhD UniversityofVirginia Departmentof ComputerScience 11/6/14 1 Wherearewe? FivemajorsecQonsofthiscourse " Regression(supervised) " ClassificaQon(supervised) " Unsupervisedmodels " Learningtheory " Graphicalmodels YanjunQi/UVACS4501L01L6501L07 11/6/14 2
2 YanjunQi/UVACS4501L01L6501L07 Announcements:RoughPlan 11/6/14 3 Announcements:RoughPlan HW5:outthisThursday,dueNov.23 rd midnight 4501$ $001:$Programmingonimagelabelingtask(similartoHW3) 6501$ $007:$Grantproposal/Businessplan If$finishing$both,$5$extra$credits$$ HW6: similartohw4 12samplesquesQonsforthepreparaQonforfinal coveringtopicsa`ermidterm Finalexam: Inclass@Thornton$Hall$E316$ Dec4 th 3:35pmL4:45pm(70mins) YanjunQi/UVACS4501L01L6501L07 11/6/14 4
3 YanjunQi/UVACS4501L01L6501L07 Announcements:RoughPlan HW3Grades+SoluQon/Willbepostedin Collabthisweekend HW4Grades/WillbepostedinCollabthis weekend MidtermGrades+SoluQon/Willbepostedin Collabthisweekend 11/6/14 5 YanjunQi/UVACS4501L01L6501L $ $007$HW5:ProjectProposal/ GrantProposal/BusinessPlan ThefollowingsecQonsareexpectedintheproposal Minimum8pages(latexandwordtemplatewillbeprovided): Abstract/Summary(300wordslimit) IntroducQonofthetargettask PrevioussoluQonforthetargettask WhyyouaretherightpersonforimplemenQngthisplan? Whymachinelearningmethodsaregoodcandidatesforthe targettask Methodyoupropose/Pleaseprovideadiagramofyour proposedsystem Experimentaldesign/detailsofthedata/wheretoget/data staqsqcs/preliminaryresults?preliminaryprototype? References(notincludedinthepagecounts) Budgetplan/Personnelplanarenotrequired 11/6/14 6
4 Wherearewe? FivemajorsecQonsofthiscourse " Regression(supervised) " ClassificaQon(supervised) " Unsupervisedmodels " Learningtheory " Graphicalmodels YanjunQi/UVACS4501L01L6501L07 11/6/14 7 hgp://scikitllearn.org/stable/tutorial/machine_learning_map/ YanjunQi/UVACS4501L01L6501L07 11/6/14 8
5 YanjunQi/UVACS4501L01L6501L07 ScikitLlearn:Regression Linearmodel figedby minimizinga regularized empiricalloss withsgd 11/6/14 9 ScikitLlearn:ClassificaQon YanjunQi/UVACS4501L01L6501L07 Tocombinethe predicqonsof severalbase esqmatorsbuilt withagiven learningalgorithm inordertoimprove generalizability/ robustnessovera singleesqmator.(1) averaging/bagging (2)boosQng approximatethe explicitfeature mappingsthat correspondto certainkernels Linearclassifiers (SVM,logisQc regression )with SGDtraining. 11/6/14 10
6 Kmeans+ GMM nexta`erclassificaqon? YanjunQi/UVACS4501L01L6501L07 Basic PCA BayesLNet HMM 11/6/14 11 Today$ YanjunQi/UVACS4501L01L6501L07 # DecisionTree(DT): # TreerepresentaQon # BriefinformaQontheory # Learningdecisiontrees # Bagging # Randomforests:EnsembleofDT # Moreaboutensemble 11/6/14 12
7 YanjunQi/UVACS4501L01L6501L07 A study comparing Classifiers Proceedingsofthe23rdInternaQonal ConferenceonMachineLearning(ICML`06). 11/6/14 13 A study comparing Classifiers 11binaryclassificaQonproblems/8metrics YanjunQi/UVACS4501L01L6501L07 11/6/14 14
8 Wherearewe? ThreemajorsecQonsforclassificaQon We can divide the large variety of classification approaches into roughly three major types 1. Discriminative - directly estimate a decision rule/boundary - e.g., logistic regression, support vector machine, decisiontree 2. Generative: - build a generative statistical model - e.g., naïve bayes classifier, Bayesian networks 3. Instance based classifiers - Use observation directly (no models) - e.g. K nearest neighbors YanjunQi/UVACS4501L01L6501L07 11/6/14 15 C3 YanjunQi/UVACS4501L01L6501L07 ADatasetfor classificaqon C3 Output as Discrete Class Label C 1, C 2,, C L Data/points/instances/examples/samples/records:[rows] Features/a0ributes/dimensions/independent3variables/covariates/predictors/regressors:[columns,exceptthelast] Target/outcome/response/label/dependent3variable:specialcolumntobepredicted[lastcolumn] 11/6/14 16
9 Example YanjunQi/UVACS4501L01L6501L07 C Example: Play Tennis 11/6/14 17 YanjunQi/UVACS4501L01L6501L07 Anatomy$of$a$decision$tree$ Outlook Each$node$is$a$test$on$$ one$feature/asribute$ sunny overcast rain Possibleagributevalues ofthenode Humidity Yes$ Windy high normal false true No$ Yes$ No$ Yes$ Leavesarethe decisions 11/6/14 18
10 YanjunQi/UVACS4501L01L6501L07 Anatomy$of$a$decision$tree$ Samplesize Outlook Each$node$is$a$test$on$$ one$asribute$ Yourdata getssmaller sunny overcast rain Possibleagributevalues ofthenode Humidity Yes$ Windy high normal false true No$ Yes$ No$ Yes$ Leavesarethe decisions 11/6/14 19 sunny ApplyModeltoTestData: To$ play$tennis $or$not.$$ Outlook overcast rain YanjunQi/UVACS4501L01L6501L07 A$new$test$example:$ (Outlook==rain)$and$ (Windy==false)$ $ Pass$it$on$the$tree$ +>$Decision$is$yes. Humidity Yes Windy high normal false true No Yes No Yes 11/6/14 20
11 sunny ApplyModeltoTestData: To$ play$tennis $or$not.$$ Outlook overcast rain YanjunQi/UVACS4501L01L6501L07 (Outlook$==overcast)$$+>$yes$ (Outlook==rain)$and$(Windy==false)$+>yes$ (Outlook==sunny)$and$(Humidity=normal)$+>yes$ Humidity Yes Windy high normal false true No Yes No Yes 11/6/14 21 Decision$trees$ DecisiontreesrepresentadisjuncQonof conjuncqonsofconstraintsontheagribute valuesofinstances. YanjunQi/UVACS4501L01L6501L07 (Outlook ==overcast) OR ((Outlook==rain) and (Windy==false)) OR ((Outlook==sunny) and (Humidity=normal)) => yes play tennis 11/6/14 22
12 RepresentaQon YanjunQi/UVACS4501L01L6501L07 0 A 1 Y=((A$and$B)$or$((not$A)$and$C))$ C B false true false true 11/6/14 23 YanjunQi/UVACS4501L01L6501L07 Sameconcept/differentrepresentaQon 0 A 1 Y=((A$and$B)$or$((not$A)$and$C))$ C B false true false true 0 C 1 B A false 1 A 0 false true true false 11/6/14 24
13 Which$aSribute$to$select$for$spliang?$ L 8+ 8L 8+ 8L 4+ 4L 4+ 4L 4+ 4L 4+ 4L 2+ 2L 2+ 2L Thisisbadsplirng thedistribuqonof eachclass(notagribute) 11/6/14 25 YanjunQi/UVACS4501L01L6501L07 How$do$we$choose$which$ asribute$to$split$?$ Whichagributeshouldbeusedasthetest? IntuiQvely,youwouldpreferthe onethatseparatesthetraining examplesasmuchaspossible. 11/6/14 26 YanjunQi/UVACS4501L01L6501L07
14 Today$ YanjunQi/UVACS4501L01L6501L07 # DecisionTree(DT): # TreerepresentaQon # BriefinformaQontheory # Learningdecisiontrees # Bagging # Randomforests:EnsembleofDT # Moreaboutensemble 11/6/14 27 InformaQongainisonecriteriato decideontheagributeforsplirng$ Imagine: 1.Someoneisabouttotellyouyourownname 2.Youareabouttoobservetheoutcomeofadiceroll 2.Youareabouttoobservetheoutcomeofacoinflip 3.Youareabouttoobservetheoutcomeofabiasedcoinflip YanjunQi/UVACS4501L01L6501L07 EachsituaQonhaveadifferentamount3of3 uncertainty3astowhatoutcomeyouwillobserve. 11/6/14 28
15 InformaQon YanjunQi/UVACS4501L01L6501L07 InformaQon: reducqoninuncertainty(amountofsurpriseintheoutcome) 1 IE ( ) = log2 log 2 px ( ) px ( ) = Iftheprobabilityofthiseventhappeningissmallandithappens, theinformaqonislarge. # Observingtheoutcomeofacoinflip ishead I = log2 1/ 2 = 1 # Observetheoutcomeofadiceis6 I = log2 1/ 6 = /6/14 29 Entropy YanjunQi/UVACS4501L01L6501L07 Theexpected3amount3of3informa9onwhenobservingthe outputofarandomvariablex H( X) = E( I( X)) = p( x) I( x) = p( x)log p( x) i i i i 2 i i IftheXcanhave8outcomesandallareequallylikely H( X ) == 1/8log 1/8= 3 i 2 11/6/14 30
16 Iftherearekpossible outcomes Entropy$ H( X) log 2 k YanjunQi/UVACS4501L01L6501L07 Equalityholdswhenall outcomesareequallylikely Themoretheprobability distribuqonthedeviates fromuniformity,thelower theentropy e.g.forarandom 11/6/14 31 binaryvariable Entropy$Lower$$beSer$purity$ Entropymeasuresthepurity YanjunQi/UVACS4501L01L6501L L 8+ 0L ThedistribuQonislessuniform Entropyislower Thenodeispurer 11/6/14 32
17 Informa8on$gain$ YanjunQi/UVACS4501L01L6501L07 IG(X_i,Y)=H(Y)LH(Y X_i) ReducQoninuncertaintybyknowingafeatureX_i InformaQongain: =(informaqonbeforesplit) (informaqona`ersplit) =entropy(parent) [averageentropy(children)] Fixed thelower,the beger(children nodesarepurer) = ForIG,the higher,the beger 11/6/14 33 Condi8onal$entropy$ H(Y ) = p( y i )log 2 p( y i ) i H(Y X ) = p(x j ) H(Y X = x j ) j = p(x j ) p( y i x j )log 2 p( y i x j ) j i YanjunQi/UVACS4501L01L6501L07 11/6/14 34
18 YanjunQi/UVACS4501L01L6501L07 Agributes Labels X1 X2 Y Count T T + 2 T F + 2 F T L 5 F F + 1 Example$ Whichonedowechoose X1orX2? IG(X1,Y) = H(Y) H(Y X1) H(Y) = - (5/10) log(5/10) -5/10log(5/10) = 1 H(Y X1) = P(X1=T)H(Y X1=T) + P(X1=F) H(Y X1=F) = 4/10 (1log log 0) +6/10 (5/6log 5/6 +1/6log1/6) = 0.39 InformaQongain(X1,Y)=1L0.39= /6/14 35 Whichonedowechoose? YanjunQi/UVACS4501L01L6501L07 X1 X2 Y Count T T + 2 T F + 2 F T L 5 F F + 1 X1 X2 Y Count T T + 2 T F + 2 F T L 5 F F + 1 Onebranch Theotherbranch Information gain (X1,Y)= 0.61 Information gain (X2,Y)= 0.12 Pick the variable which provides the most information gain about Y PickX1 $Then$recursively$choose$next$Xi$on$branches 11/6/14 36
19 Decision$Trees$ YanjunQi/UVACS4501L01L6501L07 Caveats:$$Thenumberofpossiblevaluesinfluencesthe informaqongain. Themorepossiblevalues,thehigherthegain(themorelikelyitisto formsmall,butpureparqqons) Other$Purity$(diversity)$measures InformaQonGain Gini(populaQondiversity) wherep mk isproporqonofclasskatnodem ChiLsquareTest 11/6/14 37 Overfiang$ YanjunQi/UVACS4501L01L6501L07 YoucanperfectlyfitDTtoanytrainingdata InstabilityofTrees Highvariance(smallchangesintrainingsetwill resultinchangesoftreemodel) HierarchicalstructureErrorintopsplit propagatesdown Twoapproaches: 1.Stopgrowingthetreewhenfurthersplirngthedatadoesnot yieldanimprovement 2.Growafulltree,thenprunethetree,byeliminaQngnodes. 11/6/14 38
20 FromESLbookCh9: ClassificaQonand RegressionTrees(CART) Par88on$feature$ space$into$set$of$ rectangles$ Fitsimplemodelin eachparqqon Summary:$Decision$trees$ YanjunQi/UVACS4501L01L6501L07 NonLlinearclassifier Easytouse Easytointerpret SuscepQbletooverfirngbutcanbeavoided. 11/6/14 40
21 YanjunQi/UVACS4501L01L6501L07 Today$ # DecisionTree(DT): # TreerepresentaQon # BriefinformaQontheory # Learningdecisiontrees # Bagging # Randomforests:EnsembleofDT # Moreaboutensemble 11/6/14 41 Bagging$ Baggingorbootstrap3aggrega9on33 atechniqueforreducingthevarianceofan esqmatedpredicqonfuncqon. YanjunQi/UVACS4501L01L6501L07 Forinstance,forclassificaQon,acommi0ee3 oftrees Eachtreecastsavoteforthepredictedclass. 11/6/14 42
22 Bootstrap$ YanjunQi/UVACS4501L01L6501L07 Thebasicidea: randomlydrawdatasetswith3replacement(i.e.3allows3duplicates)3 fromthetrainingdata,3eachsamplethe3same3size3as3the3original3 training3set3 3 11/6/14 43 YanjunQi/UVACS4501L01L6501L07 WithvsWithoutReplacement Bootstrapwithreplacementcankeepthe samplingsizethesameastheoriginalsizefor everyrepeatedsampling.thesampleddata groupsareindependentoneachother. Bootstrapwithoutreplacementcannotkeepthe samplingsizethesameastheoriginalsizefor everyrepeatedsampling.thesampleddata groupsaredependentoneachother. 11/6/14 44
23 YanjunQi/UVACS4501L01L6501L07 Bagging Createbootstrapsamples fromthetrainingdata Nexamples Mfeatures... $ 11/6/14 45 BaggingofDTClassifiers$ e.g. YanjunQi/UVACS4501L01L6501L07 Mfeatures Nexamples... $... $ Take$the$ majority$ vote$ i.e.refitthemodelto eachbootstrap dataset,andthen examinethebehavior overtheb replicaqons. 11/6/14 46
24 YanjunQi/UVACS4501L01L6501L07 Bagging$for$Classifica8on$with$0,1$Loss$ ClassificaQonwith0,1loss BaggingagoodclassifiercanmakeitbeSer. Baggingabadclassifiercanmakeitworse. Canunderstandthebaggingeffectintermsofaconsensusof independentweak%leaners%andwisdom%of%crowds% 11/6/14 47 Peculiari8es$ YanjunQi/UVACS4501L01L6501L07 ModelInstabilityisgoodwhenbagging$ Themorevariable(unstable)thebasicmodelis,themore improvementcanpotenqallybeobtained LowLVariabilitymethods(e.g.LDA)improvelessthanHighL Variabilitymethods(e.g.decisiontrees) LoadofRedundancy% Mostpredictorsdoroughly thesamething 11/6/14 48
25 Bagging$:$an$simulated$example N=30trainingsamples, twoclassesandp=5features, EachfeatureN(0,1)distribuQonandpairwisecorrelaQon.95 ResponseYgeneratedaccordingto: Testsamplesizeof2000 FitclassificaQontreestotrainingsetandbootstrapsamples B=200 NoQcethe bootstrap treesare differentthan theoriginal tree Fivefeatures highlycorrelated witheachother Noclear differencewith pickingupwhich featuretosplit Small changesin thetraining setwillresult indifferent tree Butthese treesare actuallyquite similarfor classificaqon
26 ForB>30,moretrees donotimprovethe baggingresults Sincethetrees correlatehighlyto eachotherandgive similarclassificaqons B Consensus:Majorityvote Probability:AveragedistribuQonatterminalnodes ESLbook/Example8.7.1 Bagging Slightlyincreasesmodelspace Cannothelpwheregreaterenlargementof spaceisneeded Baggedtreesarecorrelated UserandomforesttoreducecorrelaQon betweentrees
27 YanjunQi/UVACS4501L01L6501L07 Today$ # DecisionTree(DT): # TreerepresentaQon # BriefinformaQontheory # Learningdecisiontrees # Bagging # Randomforests:specialensembleofDT # Moreaboutensemble 11/6/14 53 Random$forest$classifier$ Randomforestclassifier, anextensiontobagging whichusesdebcorrelated3trees. YanjunQi/UVACS4501L01L6501L07 11/6/14 54
28 RandomForestClassifier Createbootstrapsamples fromthetrainingdata YanjunQi/UVACS4501L01L6501L07 Nexamples Mfeatures... $ 11/6/14 55 RandomForestClassifier YanjunQi/UVACS4501L01L6501L07 Ateachnodewhenchoosingthesplitfeature chooseonlyamongm<mfeatures Nexamples Mfeatures... $ 11/6/14 56
29 Random$Forest$Classifier$ Create$decision$tree$ from$each$bootstrap$sample$ YanjunQi/UVACS4501L01L6501L07 Nexamples Mfeatures... $... $ 11/6/14 57 Random$Forest$Classifier$ YanjunQi/UVACS4501L01L6501L07 Mfeatures Nexamples... $... $ Take$he$ majority$ vote$ 11/6/14 58
30 Random Forests 1. ForeachofourBbootstrapsamples a. Formatreeinthefollowingmanner i. Givenpdimensions,pickmofthem ii. Splitonlyaccordingtothesemdimensions 1. (wewillnotconsidertheotherpbmdimensions) iii. Repeattheabovestepsi&iiforeachsplit 1. Note:wepickadifferentsetofmdimensionsforeachsplit onasingletree YanjunQi/UVACS4501L01L6501L07 Page598L599InESLbook 11/6/14 60
31 Random Forests Randomforestcanbeviewedasarefinementofbaggingwitha tweakofdecorrela8ngthetrees: o Ateachtreesplit,arandomsubsetofmfeaturesoutofallp featuresisdrawntobeconsideredforsplirng SomeguidelinesprovidedbyBreiman,butbecarefultochoose mbasedonspecificproblem: o m=pamountstobagging o m=p/3orlog2(p)forregression o m=sqrt(p)forclassificaqon Random Forests RandomForeststrytoreducecorrelaQon betweenthetrees. Why?
32 Random Forests Assumingeachtreehasvarianceσ 2 IftreesareindependentlyidenQcally distributed,thenaveragevarianceisσ 2 /B Random Forests Assumingeachtreehasvarianceσ 2 IfsimplyidenQcallydistributed,thenaverage varianceis ρrefersto pairwise correlaqon,a posiqvevalue AsB,secondterm 0 Thus,thepairwisecorrelaQonalwaysaffectsthevariance
33 Random Forests Howtodeal? o Ifwereducem(thenumberofdimensionswe actuallyconsider), o thenwereducethepairwisetreecorrelaqon o Thus,variancewillbereduced. Today$ YanjunQi/UVACS4501L01L6501L07 # DecisionTree(DT): # TreerepresentaQon # BriefinformaQontheory # Learningdecisiontrees # Bagging # Randomforests:EnsembleofDT # Moreensemble 11/6/14 66
34 e.g. Ensembles in practice Oct2006L2009 EachraQng/sample: +<user,movie,dateofgrade,grade> Trainingset(100,480,507raQngs) Qualifyingset(2,817,131raQngs)winner Ensemble in practice Team Bellkor'sPragmaQc Chaos defeatedtheteam ensemble bysubmirng just20minutesearlier 1milliondollar TheensembleteamblendersofmulQpledifferentmethods
35 YanjunQi/UVACS4501L01L6501L07 References$$ " $Prof.Tan,Steinbach,Kumar s IntroducQon todatamining slide " HasQe,Trevor,etal.The3elements3of3 sta9s9cal3learning.vol.2.no.1.newyork: Springer,2009. " Dr.OznurTastan sslidesaboutrfanddt 11/6/14 69
UVA$CS$6316$$ $Fall$2015$Graduate:$$ Machine$Learning$$ $ $Lecture$17:$Decision(Tree(/(Random( Forest(/(Ensemble$
Dr.YanjunQi/UVACS6316/f15 UVA$CS$6316$$ $Fall$2015$Graduate:$$ Machine$Learning$$ $ $Lecture$17:$DecisionTree/Random Forest/Ensemble$ 10/28/15 Dr.YanjunQi UniversityofVirginia Departmentof ComputerScience
UVA CS 4501: Machine Learning
UVA CS 4501: Machine Learning Lecture 21: Decision Tree / Random Forest / Ensemble Dr. Yanjun Qi University of Virginia Department of Computer Science Where are we? è Five major sections of this course
Machine Learning Recitation 8 Oct 21, Oznur Tastan
Machine Learning 10601 Recitation 8 Oct 21, 2009 Oznur Tastan Outline Tree representation Brief information theory Learning decision trees Bagging Random forests Decision trees Non linear classifier Easy
Announcements:!Rough!Plan!!
UVA$CS$4501$+$001$/$6501$ $007$ Introduc8on$to$Machine$Learning$and$ Data$Mining$$ $ Lecture$26:History/Review$ YanjunQi/Jane,,PhD UniversityofVirginia Departmentof ComputerScience YanjunQi/UVACS4501O01O6501O07
Bayesian Classification. Bayesian Classification: Why?
Bayesian Classification http://css.engineering.uiowa.edu/~comp/ Bayesian Classification: Why? Probabilistic learning: Computation of explicit probabilities for hypothesis, among the most practical approaches
Learning Decision Trees
Learning Decision Trees Machine Learning Spring 2018 1 This lecture: Learning Decision Trees 1. Representation: What are decision trees? 2. Algorithm: Learning decision trees The ID3 algorithm: A greedy
The Naïve Bayes Classifier. Machine Learning Fall 2017
The Naïve Bayes Classifier Machine Learning Fall 2017 1 Today s lecture The naïve Bayes Classifier Learning the naïve Bayes Classifier Practical concerns 2 Today s lecture The naïve Bayes Classifier Learning
Decision Tree And Random Forest
Decision Tree And Random Forest Dr. Ammar Mohammed Associate Professor of Computer Science ISSR, Cairo University PhD of CS ( Uni. Koblenz-Landau, Germany) Spring 2019 Contact: mailto: Ammar@cu.edu.eg
Decision Trees. Nicholas Ruozzi University of Texas at Dallas. Based on the slides of Vibhav Gogate and David Sontag
Decision Trees Nicholas Ruozzi University of Texas at Dallas Based on the slides of Vibhav Gogate and David Sontag Supervised Learning Input: labelled training data i.e., data plus desired output Assumption:
UVA$CS$6316$$ $Fall$2015$Graduate:$$ Machine$Learning$$ $ $Lecture$8:$Supervised$ClassificaDon$ with$support$vector$machine$
UVACS6316 Fall2015Graduate: MachineLearning Lecture8:SupervisedClassificaDon withsupportvectormachine 9/30/15 Dr.YanjunQi UniversityofVirginia Departmentof ComputerScience 1 Whereweare?! FivemajorsecHonsofthiscourse
Decision Trees. Data Science: Jordan Boyd-Graber University of Maryland MARCH 11, Data Science: Jordan Boyd-Graber UMD Decision Trees 1 / 1
Decision Trees Data Science: Jordan Boyd-Graber University of Maryland MARCH 11, 2018 Data Science: Jordan Boyd-Graber UMD Decision Trees 1 / 1 Roadmap Classification: machines labeling data for us Last
( D) I(2,3) I(4,0) I(3,2) weighted avg. of entropies
Decision Tree Induction using Information Gain Let I(x,y) as the entropy in a dataset with x number of class 1(i.e., play ) and y number of class (i.e., don t play outcomes. The entropy at the root, i.e.,
Learning Decision Trees
Learning Decision Trees Machine Learning Fall 2018 Some slides from Tom Mitchell, Dan Roth and others 1 Key issues in machine learning Modeling How to formulate your problem as a machine learning problem?
Bayesian Learning. Reading: Tom Mitchell, Generative and discriminative classifiers: Naive Bayes and logistic regression, Sections 1-2.
Bayesian Learning Reading: Tom Mitchell, Generative and discriminative classifiers: Naive Bayes and logistic regression, Sections 1-2. (Linked from class website) Conditional Probability Probability of
Machine Learning 2nd Edi7on
Lecture Slides for INTRODUCTION TO Machine Learning 2nd Edi7on CHAPTER 9: Decision Trees ETHEM ALPAYDIN The MIT Press, 2010 Edited and expanded for CS 4641 by Chris Simpkins alpaydin@boun.edu.tr h1p://www.cmpe.boun.edu.tr/~ethem/i2ml2e
Numerical Learning Algorithms
Numerical Learning Algorithms Example SVM for Separable Examples.......................... Example SVM for Nonseparable Examples....................... 4 Example Gaussian Kernel SVM...............................
Learning Classification Trees. Sargur Srihari
Learning Classification Trees Sargur srihari@cedar.buffalo.edu 1 Topics in CART CART as an adaptive basis function model Classification and Regression Tree Basics Growing a Tree 2 A Classification Tree
Classification and Regression Trees
Classification and Regression Trees Ryan P Adams So far, we have primarily examined linear classifiers and regressors, and considered several different ways to train them When we ve found the linearity
UVA CS / Introduc8on to Machine Learning and Data Mining
UVA CS 4501-001 / 6501 007 Introduc8on to Machine Learning and Data Mining Lecture 13: Probability and Sta3s3cs Review (cont.) + Naïve Bayes Classifier Yanjun Qi / Jane, PhD University of Virginia Department
Decision Tree Learning and Inductive Inference
Decision Tree Learning and Inductive Inference 1 Widely used method for inductive inference Inductive Inference Hypothesis: Any hypothesis found to approximate the target function well over a sufficiently
Bayesian Learning. Artificial Intelligence Programming. 15-0: Learning vs. Deduction
15-0: Learning vs. Deduction Artificial Intelligence Programming Bayesian Learning Chris Brooks Department of Computer Science University of San Francisco So far, we ve seen two types of reasoning: Deductive
Introduction. Decision Tree Learning. Outline. Decision Tree 9/7/2017. Decision Tree Definition
Introduction Decision Tree Learning Practical methods for inductive inference Approximating discrete-valued functions Robust to noisy data and capable of learning disjunctive expression ID3 earch a completely
Mining Classification Knowledge
Mining Classification Knowledge Remarks on NonSymbolic Methods JERZY STEFANOWSKI Institute of Computing Sciences, Poznań University of Technology COST Doctoral School, Troina 2008 Outline 1. Bayesian classification
Lecture 24: Other (Non-linear) Classifiers: Decision Tree Learning, Boosting, and Support Vector Classification Instructor: Prof. Ganesh Ramakrishnan
Lecture 24: Other (Non-linear) Classifiers: Decision Tree Learning, Boosting, and Support Vector Classification Instructor: Prof Ganesh Ramakrishnan October 20, 2016 1 / 25 Decision Trees: Cascade of step
LINEAR CLASSIFICATION, PERCEPTRON, LOGISTIC REGRESSION, SVC, NAÏVE BAYES. Supervised Learning
LINEAR CLASSIFICATION, PERCEPTRON, LOGISTIC REGRESSION, SVC, NAÏVE BAYES Supervised Learning Linear vs non linear classifiers In K-NN we saw an example of a non-linear classifier: the decision boundary
Artificial Intelligence. Topic
Artificial Intelligence Topic What is decision tree? A tree where each branching node represents a choice between two or more alternatives, with every branching node being part of a path to a leaf node
Machine Learning. Yuh-Jye Lee. March 1, Lab of Data Science and Machine Intelligence Dept. of Applied Math. at NCTU
Machine Learning Yuh-Jye Lee Lab of Data Science and Machine Intelligence Dept. of Applied Math. at NCTU March 1, 2017 1 / 13 Bayes Rule Bayes Rule Assume that {B 1, B 2,..., B k } is a partition of S
Imagine we ve got a set of data containing several types, or classes. E.g. information about customers, and class=whether or not they buy anything.
Decision Trees Defining the Task Imagine we ve got a set of data containing several types, or classes. E.g. information about customers, and class=whether or not they buy anything. Can we predict, i.e
Algorithms for Classification: The Basic Methods
Algorithms for Classification: The Basic Methods Outline Simplicity first: 1R Naïve Bayes 2 Classification Task: Given a set of pre-classified examples, build a model or classifier to classify new cases.
Mining Classification Knowledge
Mining Classification Knowledge Remarks on NonSymbolic Methods JERZY STEFANOWSKI Institute of Computing Sciences, Poznań University of Technology SE lecture revision 2013 Outline 1. Bayesian classification
Classification. Classification. What is classification. Simple methods for classification. Classification by decision tree induction
Classification What is classification Classification Simple methods for classification Classification by decision tree induction Classification evaluation Classification in Large Databases Classification
COMP 328: Machine Learning
COMP 328: Machine Learning Lecture 2: Naive Bayes Classifiers Nevin L. Zhang Department of Computer Science and Engineering The Hong Kong University of Science and Technology Spring 2010 Nevin L. Zhang
The Solution to Assignment 6
The Solution to Assignment 6 Problem 1: Use the 2-fold cross-validation to evaluate the Decision Tree Model for trees up to 2 levels deep (that is, the maximum path length from the root to the leaves is
Introduction to ML. Two examples of Learners: Naïve Bayesian Classifiers Decision Trees
Introduction to ML Two examples of Learners: Naïve Bayesian Classifiers Decision Trees Why Bayesian learning? Probabilistic learning: Calculate explicit probabilities for hypothesis, among the most practical
Ensemble Methods. Charles Sutton Data Mining and Exploration Spring Friday, 27 January 12
Ensemble Methods Charles Sutton Data Mining and Exploration Spring 2012 Bias and Variance Consider a regression problem Y = f(x)+ N(0, 2 ) With an estimate regression function ˆf, e.g., ˆf(x) =w > x Suppose
Induction on Decision Trees
Séance «IDT» de l'ue «apprentissage automatique» Bruno Bouzy bruno.bouzy@parisdescartes.fr www.mi.parisdescartes.fr/~bouzy Outline Induction task ID3 Entropy (disorder) minimization Noise Unknown attribute
Introduction to Machine Learning CMU-10701
Introduction to Machine Learning CMU-10701 23. Decision Trees Barnabás Póczos Contents Decision Trees: Definition + Motivation Algorithm for Learning Decision Trees Entropy, Mutual Information, Information
Decision Trees. Danushka Bollegala
Decision Trees Danushka Bollegala Rule-based Classifiers In rule-based learning, the idea is to learn a rule from train data in the form IF X THEN Y (or a combination of nested conditions) that explains
Decision Tree Learning Mitchell, Chapter 3. CptS 570 Machine Learning School of EECS Washington State University
Decision Tree Learning Mitchell, Chapter 3 CptS 570 Machine Learning School of EECS Washington State University Outline Decision tree representation ID3 learning algorithm Entropy and information gain
Bayesian Learning. Bayesian Learning Criteria
Bayesian Learning In Bayesian learning, we are interested in the probability of a hypothesis h given the dataset D. By Bayes theorem: P (h D) = P (D h)p (h) P (D) Other useful formulas to remember are:
FINAL: CS 6375 (Machine Learning) Fall 2014
FINAL: CS 6375 (Machine Learning) Fall 2014 The exam is closed book. You are allowed a one-page cheat sheet. Answer the questions in the spaces provided on the question sheets. If you run out of room for
Inteligência Artificial (SI 214) Aula 15 Algoritmo 1R e Classificador Bayesiano
Inteligência Artificial (SI 214) Aula 15 Algoritmo 1R e Classificador Bayesiano Prof. Josenildo Silva jcsilva@ifma.edu.br 2015 2012-2015 Josenildo Silva (jcsilva@ifma.edu.br) Este material é derivado dos
Decision Tree Learning
Topics Decision Tree Learning Sattiraju Prabhakar CS898O: DTL Wichita State University What are decision trees? How do we use them? New Learning Task ID3 Algorithm Weka Demo C4.5 Algorithm Weka Demo Implementation
ARTIFICIAL INTELLIGENCE. Supervised learning: classification
INFOB2KI 2017-2018 Utrecht University The Netherlands ARTIFICIAL INTELLIGENCE Supervised learning: classification Lecturer: Silja Renooij These slides are part of the INFOB2KI Course Notes available from
Data Mining Classification: Basic Concepts, Decision Trees, and Model Evaluation
Data Mining Classification: Basic Concepts, Decision Trees, and Model Evaluation Lecture Notes for Chapter 4 Part I Introduction to Data Mining by Tan, Steinbach, Kumar Adapted by Qiang Yang (2010) Tan,Steinbach,
Machine learning comes from Bayesian decision theory in statistics. There we want to minimize the expected value of the loss function.
Bayesian learning: Machine learning comes from Bayesian decision theory in statistics. There we want to minimize the expected value of the loss function. Let y be the true label and y be the predicted
Chapter 6: Classification
Chapter 6: Classification 1) Introduction Classification problem, evaluation of classifiers, prediction 2) Bayesian Classifiers Bayes classifier, naive Bayes classifier, applications 3) Linear discriminant
the tree till a class assignment is reached
Decision Trees Decision Tree for Playing Tennis Prediction is done by sending the example down Prediction is done by sending the example down the tree till a class assignment is reached Definitions Internal
CS6375: Machine Learning Gautam Kunapuli. Decision Trees
Gautam Kunapuli Example: Restaurant Recommendation Example: Develop a model to recommend restaurants to users depending on their past dining experiences. Here, the features are cost (x ) and the user s
Classification II: Decision Trees and SVMs
Classification II: Decision Trees and SVMs Digging into Data: Jordan Boyd-Graber February 25, 2013 Slides adapted from Tom Mitchell, Eric Xing, and Lauren Hannah Digging into Data: Jordan Boyd-Graber ()
UVA$CS$6316$$ $Fall$2015$Graduate:$$ Machine$Learning$$ $ $Lecture$4:$More$opBmizaBon$for$Linear$ Regression$
Dr.YanjunQi/UVACS6316/f15 UVACS6316 Fall2015Graduate: MachineLearning Lecture4:MoreopBmizaBonforLinear Regression 9/14/15 Dr.YanjunQi UniversityofVirginia Departmentof ComputerScience 1 Whereweare?! FivemajorsecHonsofthiscourse
Dan Roth 461C, 3401 Walnut
CIS 519/419 Applied Machine Learning www.seas.upenn.edu/~cis519 Dan Roth danroth@seas.upenn.edu http://www.cis.upenn.edu/~danroth/ 461C, 3401 Walnut Slides were created by Dan Roth (for CIS519/419 at Penn
Classification CE-717: Machine Learning Sharif University of Technology. M. Soleymani Fall 2012
Classification CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall 2012 Topics Discriminant functions Logistic regression Perceptron Generative models Generative vs. discriminative
EECS 349:Machine Learning Bryan Pardo
EECS 349:Machine Learning Bryan Pardo Topic 2: Decision Trees (Includes content provided by: Russel & Norvig, D. Downie, P. Domingos) 1 General Learning Task There is a set of possible examples Each example
The Bayesian Learning
The Bayesian Learning Rodrigo Fernandes de Mello Invited Professor at Télécom ParisTech Associate Professor at Universidade de São Paulo, ICMC, Brazil http://www.icmc.usp.br/~mello mello@icmc.usp.br First
Variance Reduction and Ensemble Methods
Variance Reduction and Ensemble Methods Nicholas Ruozzi University of Texas at Dallas Based on the slides of Vibhav Gogate and David Sontag Last Time PAC learning Bias/variance tradeoff small hypothesis
Bayesian Learning Features of Bayesian learning methods:
Bayesian Learning Features of Bayesian learning methods: Each observed training example can incrementally decrease or increase the estimated probability that a hypothesis is correct. This provides a more
Decision Trees.
. Machine Learning Decision Trees Prof. Dr. Martin Riedmiller AG Maschinelles Lernen und Natürlichsprachliche Systeme Institut für Informatik Technische Fakultät Albert-Ludwigs-Universität Freiburg riedmiller@informatik.uni-freiburg.de
Classification and regression trees
Classification and regression trees Pierre Geurts p.geurts@ulg.ac.be Last update: 23/09/2015 1 Outline Supervised learning Decision tree representation Decision tree learning Extensions Regression trees
Classification Lecture 2: Methods
Classification Lecture 2: Methods Jing Gao SUNY Buffalo 1 Outline Basics Problem, goal, evaluation Methods Nearest Neighbor Decision Tree Naïve Bayes Rule-based Classification Logistic Regression Support
Bayes Classifiers. CAP5610 Machine Learning Instructor: Guo-Jun QI
Bayes Classifiers CAP5610 Machine Learning Instructor: Guo-Jun QI Recap: Joint distributions Joint distribution over Input vector X = (X 1, X 2 ) X 1 =B or B (drinking beer or not) X 2 = H or H (headache
Decision Trees. Gavin Brown
Decision Trees Gavin Brown Every Learning Method has Limitations Linear model? KNN? SVM? Explain your decisions Sometimes we need interpretable results from our techniques. How do you explain the above
Decision Tree Analysis for Classification Problems. Entscheidungsunterstützungssysteme SS 18
Decision Tree Analysis for Classification Problems Entscheidungsunterstützungssysteme SS 18 Supervised segmentation An intuitive way of thinking about extracting patterns from data in a supervised manner
ECE 5984: Introduction to Machine Learning
ECE 5984: Introduction to Machine Learning Topics: Ensemble Methods: Bagging, Boosting Readings: Murphy 16.4; Hastie 16 Dhruv Batra Virginia Tech Administrativia HW3 Due: April 14, 11:55pm You will implement
CS 446 Machine Learning Fall 2016 Nov 01, Bayesian Learning
CS 446 Machine Learning Fall 206 Nov 0, 206 Bayesian Learning Professor: Dan Roth Scribe: Ben Zhou, C. Cervantes Overview Bayesian Learning Naive Bayes Logistic Regression Bayesian Learning So far, we
Computational Genomics
Computational Genomics http://www.cs.cmu.edu/~02710 Introduction to probability, statistics and algorithms (brief) intro to probability Basic notations Random variable - referring to an element / event
UVA$CS$6316$$ $Fall$2015$Graduate:$$ Machine$Learning$$ $ $Lecture$18:$Neural'Network'/'Deep' Learning$
UVA$CS$6316$$ $Fall$2015$Graduate:$$ Machine$Learning$$ $ $Lecture$18:$NeuralNetwork/Deep Learning$ Dr.YanjunQi UniversityofVirginia Departmentof ComputerScience 1 Wherearewe?! FivemajorsecKonsofthiscourse
UVA$CS$6316$$ $Fall$2015$Graduate:$$ Machine$Learning$$ $ $Lecture$5:$NonALinear$Regression$ Models$
UVACS6316 Fall15Graduate: MachineLearning Lecture5:NonALinearRegression Models 9/1/15 Dr.YanjunQi UniversityofVirginia Departmentof ComputerScience 1 Whereweare?! FivemajorsecHonsofthiscourse " Regression(supervised)
Decision Trees. Tirgul 5
Decision Trees Tirgul 5 Using Decision Trees It could be difficult to decide which pet is right for you. We ll find a nice algorithm to help us decide what to choose without having to think about it. 2
Decision Tree Learning - ID3
Decision Tree Learning - ID3 n Decision tree examples n ID3 algorithm n Occam Razor n Top-Down Induction in Decision Trees n Information Theory n gain from property 1 Training Examples Day Outlook Temp.
Decision Trees.
. Machine Learning Decision Trees Prof. Dr. Martin Riedmiller AG Maschinelles Lernen und Natürlichsprachliche Systeme Institut für Informatik Technische Fakultät Albert-Ludwigs-Universität Freiburg riedmiller@informatik.uni-freiburg.de
Data classification (II)
Lecture 4: Data classification (II) Data Mining - Lecture 4 (2016) 1 Outline Decision trees Choice of the splitting attribute ID3 C4.5 Classification rules Covering algorithms Naïve Bayes Classification
Naive Bayes Classifier. Danushka Bollegala
Naive Bayes Classifier Danushka Bollegala Bayes Rule The probability of hypothesis H, given evidence E P(H E) = P(E H)P(H)/P(E) Terminology P(E): Marginal probability of the evidence E P(H): Prior probability
CSCE 478/878 Lecture 6: Bayesian Learning and Graphical Models. Stephen Scott. Introduction. Outline. Bayes Theorem. Formulas
ian ian ian Might have reasons (domain information) to favor some hypotheses/predictions over others a priori ian methods work with probabilities, and have two main roles: Naïve Nets (Adapted from Ethem
Stephen Scott.
1 / 28 ian ian Optimal (Adapted from Ethem Alpaydin and Tom Mitchell) Naïve Nets sscott@cse.unl.edu 2 / 28 ian Optimal Naïve Nets Might have reasons (domain information) to favor some hypotheses/predictions
Supervised Learning: Regression & Classifiers. Fall 2010
Supervised Learning: Regression & Classifiers Fall 2010 Problem Statement Given training data of the form: {( xi, yi)...( xm, ym)} X: the space of input features/attributes Y: the space of output values
10-701/ Machine Learning: Assignment 1
10-701/15-781 Machine Learning: Assignment 1 The assignment is due September 27, 2005 at the beginning of class. Write your name in the top right-hand corner of each page submitted. No paperclips, folders,
Intuition Bayesian Classification
Intuition Bayesian Classification More ockey fans in Canada tan in US Wic country is Tom, a ockey ball fan, from? Predicting Canada as a better cance to be rigt Prior probability P(Canadian=5%: reflect
UVA CS 6316/4501 Fall 2016 Machine Learning. Lecture 11: Probability Review. Dr. Yanjun Qi. University of Virginia. Department of Computer Science
UVA CS 6316/4501 Fall 2016 Machine Learning Lecture 11: Probability Review 10/17/16 Dr. Yanjun Qi University of Virginia Department of Computer Science 1 Announcements: Schedule Midterm Nov. 26 Wed / 3:30pm
CSCE 478/878 Lecture 6: Bayesian Learning
Bayesian Methods Not all hypotheses are created equal (even if they are all consistent with the training data) Outline CSCE 478/878 Lecture 6: Bayesian Learning Stephen D. Scott (Adapted from Tom Mitchell
Where are we? è Five major sec3ons of this course
UVA CS 4501-001 / 6501 007 Introduc8on to Machine Learning and Data Mining Lecture 12: Probability and Sta3s3cs Review Yanjun Qi / Jane University of Virginia Department of Computer Science 10/02/14 1
CS145: INTRODUCTION TO DATA MINING
CS145: INTRODUCTION TO DATA MINING 4: Vector Data: Decision Tree Instructor: Yizhou Sun yzsun@cs.ucla.edu October 10, 2017 Methods to Learn Vector Data Set Data Sequence Data Text Data Classification Clustering
Lazy Rule Learning Nikolaus Korfhage
Lazy Rule Learning Nikolaus Korfhage 19. Januar 2012 TU-Darmstadt Nikolaus Korfhage 1 Introduction Lazy Rule Learning Algorithm Possible Improvements Improved Lazy Rule Learning Algorithm Implementation
Data Mining Part 4. Prediction
Data Mining Part 4. Prediction 4.3. Fall 2009 Instructor: Dr. Masoud Yaghini Outline Introduction Bayes Theorem Naïve References Introduction Bayesian classifiers A statistical classifiers Introduction
Topics. Bayesian Learning. What is Bayesian Learning? Objectives for Bayesian Learning
Topics Bayesian Learning Sattiraju Prabhakar CS898O: ML Wichita State University Objectives for Bayesian Learning Bayes Theorem and MAP Bayes Optimal Classifier Naïve Bayes Classifier An Example Classifying
Administrative notes. Computational Thinking ct.cs.ubc.ca
Administrative notes Labs this week: project time. Remember, you need to pass the project in order to pass the course! (See course syllabus.) Clicker grades should be on-line now Administrative notes March
Decision trees. Special Course in Computer and Information Science II. Adam Gyenge Helsinki University of Technology
Decision trees Special Course in Computer and Information Science II Adam Gyenge Helsinki University of Technology 6.2.2008 Introduction Outline: Definition of decision trees ID3 Pruning methods Bibliography:
Supervised Learning! Algorithm Implementations! Inferring Rudimentary Rules and Decision Trees!
Supervised Learning! Algorithm Implementations! Inferring Rudimentary Rules and Decision Trees! Summary! Input Knowledge representation! Preparing data for learning! Input: Concept, Instances, Attributes"
Supervised Learning (contd) Decision Trees. Mausam (based on slides by UW-AI faculty)
Supervised Learning (contd) Decision Trees Mausam (based on slides by UW-AI faculty) Decision Trees To play or not to play? http://www.sfgate.com/blogs/images/sfgate/sgreen/2007/09/05/2240773250x321.jpg
Artificial Intelligence Decision Trees
Artificial Intelligence Decision Trees Andrea Torsello Decision Trees Complex decisions can often be expressed in terms of a series of questions: What to do this Weekend? If my parents are visiting We
Data Mining und Maschinelles Lernen
Data Mining und Maschinelles Lernen Ensemble Methods Bias-Variance Trade-off Basic Idea of Ensembles Bagging Basic Algorithm Bagging with Costs Randomization Random Forests Boosting Stacking Error-Correcting
COMP61011! Probabilistic Classifiers! Part 1, Bayes Theorem!
COMP61011 Probabilistic Classifiers Part 1, Bayes Theorem Reverend Thomas Bayes, 1702-1761 p ( T W ) W T ) T ) W ) Bayes Theorem forms the backbone of the past 20 years of ML research into probabilistic
Rule Generation using Decision Trees
Rule Generation using Decision Trees Dr. Rajni Jain 1. Introduction A DT is a classification scheme which generates a tree and a set of rules, representing the model of different classes, from a given
Lecture 3: Decision Trees
Lecture 3: Decision Trees Cognitive Systems - Machine Learning Part I: Basic Approaches of Concept Learning ID3, Information Gain, Overfitting, Pruning last change November 26, 2014 Ute Schmid (CogSys,
CS7267 MACHINE LEARNING
CS7267 MACHINE LEARNING ENSEMBLE LEARNING Ref: Dr. Ricardo Gutierrez-Osuna at TAMU, and Aarti Singh at CMU Mingon Kang, Ph.D. Computer Science, Kennesaw State University Definition of Ensemble Learning
Decision Trees. CSC411/2515: Machine Learning and Data Mining, Winter 2018 Luke Zettlemoyer, Carlos Guestrin, and Andrew Moore
Decision Trees Claude Monet, The Mulberry Tree Slides from Pedro Domingos, CSC411/2515: Machine Learning and Data Mining, Winter 2018 Luke Zettlemoyer, Carlos Guestrin, and Andrew Moore Michael Guerzhoy
Classification: Decision Trees
Classification: Decision Trees Outline Top-Down Decision Tree Construction Choosing the Splitting Attribute Information Gain and Gain Ratio 2 DECISION TREE An internal node is a test on an attribute. A
Introduction to Machine Learning Midterm, Tues April 8
Introduction to Machine Learning 10-701 Midterm, Tues April 8 [1 point] Name: Andrew ID: Instructions: You are allowed a (two-sided) sheet of notes. Exam ends at 2:45pm Take a deep breath and don t spend
Midterm Review CS 6375: Machine Learning. Vibhav Gogate The University of Texas at Dallas
Midterm Review CS 6375: Machine Learning Vibhav Gogate The University of Texas at Dallas Machine Learning Supervised Learning Unsupervised Learning Reinforcement Learning Parametric Y Continuous Non-parametric
MIDTERM: CS 6375 INSTRUCTOR: VIBHAV GOGATE October,
MIDTERM: CS 6375 INSTRUCTOR: VIBHAV GOGATE October, 23 2013 The exam is closed book. You are allowed a one-page cheat sheet. Answer the questions in the spaces provided on the question sheets. If you run