26 Chapter 4 Classification
|
|
|
- Rodger Flowers
- 5 years ago
- Views:
Transcription
1 26 Chapter 4 Classification The preceding tree cannot be simplified. 2. Consider the training examples shown in Table 4.1 for a binary classification problem. Table 4.1. Data set for Exercise 2. Customer ID Gender Car Type Shirt Size Class 1 M Family Small C0 2 M Sports Medium C0 3 M Sports Medium C0 4 M Sports Large C0 5 M Sports Extra Large C0 6 M Sports Extra Large C0 7 F Sports Small C0 8 F Sports Small C0 F Sports Medium C0 10 F Luxury Large C0 11 M Family Large C1 12 M Family Extra Large C1 13 M Family Medium C1 14 M Luxury Extra Large C1 15 F Luxury Small C1 16 F Luxury Small C1 17 F Luxury Medium C1 18 F Luxury Medium C1 1 F Luxury Medium C1 20 F Luxury Large C1 (a) Compute the Gini index for the overall collection of training examples. Gini = =0.5. (b) Compute the Gini index for the Customer ID attribute. The gini for each Customer ID value is 0. Therefore, the overall gini for Customer ID is 0. (c) Compute the Gini index for the Gender attribute. The gini for Male is =0.5. The gini for Female is also 0.5. Therefore, the overall gini for Gender is =0.5.
2 27 Table 4.2. Data set for Exercise 3. Instance a 1 a 2 a 3 Target Class 1 T T T T T F F F F T F T F F T F F T 5.0 (d) Compute the Gini index for the Car Type attribute using multiway split. The gini for Family car is 0.375, Sports car is 0, and Luxury car is The overall gini is (e) Compute the Gini index for the Shirt Size attribute using multiway split. The gini for Small shirt size is 0.48, Medium shirt size is 0.488, Large shirt size is 0.5, and Extra Large shirt size is 0.5. The overall gini for Shirt Size attribute is (f) Which attribute is better, Gender, Car Type, orshirt Size? Car Type because it has the lowest gini among the three attributes. (g) Explain why Customer ID should not be used as the attribute test condition even though it has the lowest Gini. The attribute has no predictive power since new customers are assigned to new Customer IDs. 3. Consider the training examples shown in Table 4.2 for a binary classification problem. (a) What is the entropy of this collection of training examples with respect to the positive class? There are four positive examples and five negative examples. Thus, P (+) = 4/ andp ( ) =5/. The entropy of the training examples is 4/ log 2 (4/) 5/ log 2 (5/) = 0.11.
3 28 Chapter 4 Classification (b) What are the information gains of a 1 and a 2 relative to these training examples? For attribute a 1, the corresponding counts and probabilities are: a T 3 1 F 1 4 The entropy for a 1 is [ 4 (3/4) log 2 (3/4) (1/4) log 2 (1/4) + 5 [ (1/5) log 2 (1/5) (4/5) log 2 (4/5) = Therefore, the information gain for a 1 is = For attribute a 2, the corresponding counts and probabilities are: a T 2 3 F 2 2 The entropy for a 2 is [ 5 (2/5) log 2 (2/5) (3/5) log 2 (3/5) + 4 [ (2/4) log 2 (2/4) (2/4) log 2 (2/4) =0.83. Therefore, the information gain for a 2 is = (c) For a 3, which is a continuous attribute, compute the information gain for every possible split. a 3 Class label Split point Entropy Info Gain The best split for a 3 occurs at split point equals to 2.
4 2 (d) What is the best split (among a 1, a 2,anda 3 )accordingtotheinformation gain? According to information gain, a 1 produces the best split. (e) What is the best split (between a 1 and a 2 ) according to the classification error rate? For attribute a 1 : error rate = 2/. For attribute a 2 : error rate = 4/. Therefore, according to error rate, a 1 produces the best split. (f) What is the best split (between a 1 and a 2 ) according to the Gini index? For attribute a 1, the gini index is 4 [1 (3/4) 2 (1/4) [1 (1/5) 2 (4/5) 2 = For attribute a 2, the gini index is 5 [1 (2/5) 2 (3/5) [1 (2/4) 2 (2/4) 2 = Since the gini index for a 1 is smaller, it produces the better split. 4. Show that the entropy of a node never increases after splitting it into smaller successor nodes. Let Y = {y 1,y 2,,y c } denote the c classes and X = {x 1,x 2,,x k } denote the k attribute values of an attribute X. Before a node is split on X, the entropy is: E(Y )= P (y j ) log 2 P (y j )= i=1 P (x i,y j ) log 2 P (y j ), (4.1) where we have used the fact that P (y j )= k i=1 P (x i,y j ) from the law of total probability. After splitting on X, the entropy for each child node X = x i is: E(Y x i )= P (y j x i ) log 2 P (y j x i ) (4.2)
5 30 Chapter 4 Classification where P (y j x i ) is the fraction of examples with X = x i that belong to class y j. The entropy after splitting on X is given by the weighted entropy of the children nodes: E(Y X) = P (x i )E(Y x i ) i=1 = = P (x i )P (y j x i ) log 2 P (y j x i ) P (x i,y j ) log 2 P (y j x i ), (4.3) where we have used a known fact from probability theory that P (x i,y j )= P (y j x i ) P (x i ). Note that E(Y X) is also known as the conditional entropy of Y given X. To answer this question, we need to show that E(Y X) E(Y ). Let us compute the difference between the entropies after splitting and before splitting, i.e., E(Y X) E(Y ), using Equations 4.1 and 4.3: E(Y X) E(Y ) = P (x i,y j ) log 2 P (y j x i )+ = = P (x i,y j ) log 2 P (y j ) P (y j x i ) P (x i,y j ) log 2 P (x i )P (y j ) P (x i,y j ) P (x i,y j ) log 2 P (y j ) (4.4) To prove that Equation 4.4 is non-positive, we use the following property of a logarithmic function: d a k log(z k ) log ( d ) a k z k, (4.5) k=1 subject to the condition that d k=1 a k = 1. This property is a special case of a more general theorem involving convex functions (which include the logarithmic function) known as Jensen s inequality. k=1
6 By applying Jensen s inequality, Equation 4.4 can be bounded as follows: E(Y X) E(Y ) log 2 [ i=1 = log 2 [ i=1 P (x i ) = log 2 (1) = 0 P (x i,y j ) P (x i)p (y j ) P (x i,y j ) P (y j ) Because E(Y X) E(Y ) 0, it follows that entropy never increases after splitting on an attribute. 5. Consider the following data set for a binary class problem. A B Class Label T F + T T + T T + T F T T + F F F F F F T T T F (a) Calculate the information gain when splitting on A and B. Which attribute would the decision tree induction algorithm choose? The contingency tables after splitting on attributes A and B are: A = T A = F B = T B = F The overall entropy before splitting is: E orig = 0.4 log log 0.6 =0.710 The information gain after splitting on A is: E A=T = 4 7 log log 3 7 =0.852 E A=F = 3 3 log log 0 3 =0 = E orig 7/10E A=T 3/10E A=F =
7 32 Chapter 4 Classification The information gain after splitting on B is: E B=T = 3 4 log log 1 4 = E B=F = 1 6 log log 5 6 = = E orig 4/10E B=T 6/10E B=F = Therefore, attribute A will be chosen to split the node. (b) Calculate the gain in the Gini index when splitting on A and B. Which attribute would the decision tree induction algorithm choose? The overall gini before splitting is: G orig = =0.48 The gain in gini after splitting on A is: ( ) 2 ( ) G A=T = 1 = ( ) 2 ( ) G A=F = 1 = =0 3 3 = G orig 7/10G A=T 3/10G A=F = The gain in gini after splitting on B is: ( ) 2 ( ) G B=T = 1 = ( ) 2 ( ) G B=F = 1 = = = G orig 4/10G B=T 6/10G B=F = Therefore, attribute B will be chosen to split the node. (c) Figure 4.13 shows that entropy and the Gini index are both monotonously increasing on the range [0, 0.5 and they are both monotonously decreasing on the range [0.5, 1. Is it possible that information gain and the gain in the Gini index favor different attributes? Explain. Yes, even though these measures have similar range and monotonous behavior, their respective gains,, which are scaled differences of the measures, do not necessarily behave in the same way, as illustrated by the results in parts (a) and (b). 6. Consider the following set of training examples.
Lecture VII: Classification I. Dr. Ouiem Bchir
Lecture VII: Classification I Dr. Ouiem Bchir 1 Classification: Definition Given a collection of records (training set ) Each record contains a set of attributes, one of the attributes is the class. Find
Review of Lecture 1. Across records. Within records. Classification, Clustering, Outlier detection. Associations
Review of Lecture 1 This course is about finding novel actionable patterns in data. We can divide data mining algorithms (and the patterns they find) into five groups Across records Classification, Clustering,
Q1 (12 points): Chap 4 Exercise 3 (a) to (f) (2 points each)
Q1 (1 points): Chap 4 Exercise 3 (a) to (f) ( points each) Given a table Table 1 Dataset for Exercise 3 Instance a 1 a a 3 Target Class 1 T T 1.0 + T T 6.0 + 3 T F 5.0-4 F F 4.0 + 5 F T 7.0-6 F T 3.0-7
Data Mining Classification: Basic Concepts and Techniques. Lecture Notes for Chapter 3. Introduction to Data Mining, 2nd Edition
Data Mining Classification: Basic Concepts and Techniques Lecture Notes for Chapter 3 by Tan, Steinbach, Karpatne, Kumar 1 Classification: Definition Given a collection of records (training set ) Each
the tree till a class assignment is reached
Decision Trees Decision Tree for Playing Tennis Prediction is done by sending the example down Prediction is done by sending the example down the tree till a class assignment is reached Definitions Internal
Lecture 7 Decision Tree Classifier
Machine Learning Dr.Ammar Mohammed Lecture 7 Decision Tree Classifier Decision Tree A decision tree is a simple classifier in the form of a hierarchical tree structure, which performs supervised classification
Data Mining and Analysis: Fundamental Concepts and Algorithms
Data Mining and Analysis: Fundamental Concepts and Algorithms dataminingbook.info Mohammed J. Zaki 1 Wagner Meira Jr. 2 1 Department of Computer Science Rensselaer Polytechnic Institute, Troy, NY, USA
EECS 349:Machine Learning Bryan Pardo
EECS 349:Machine Learning Bryan Pardo Topic 2: Decision Trees (Includes content provided by: Russel & Norvig, D. Downie, P. Domingos) 1 General Learning Task There is a set of possible examples Each example
Decision Trees. CS57300 Data Mining Fall Instructor: Bruno Ribeiro
Decision Trees CS57300 Data Mining Fall 2016 Instructor: Bruno Ribeiro Goal } Classification without Models Well, partially without a model } Today: Decision Trees 2015 Bruno Ribeiro 2 3 Why Trees? } interpretable/intuitive,
Randomized Decision Trees
Randomized Decision Trees compiled by Alvin Wan from Professor Jitendra Malik s lecture Discrete Variables First, let us consider some terminology. We have primarily been dealing with real-valued data,
Decision T ree Tree Algorithm Week 4 1
Decision Tree Algorithm Week 4 1 Team Homework Assignment #5 Read pp. 105 117 of the text book. Do Examples 3.1, 3.2, 3.3 and Exercise 3.4 (a). Prepare for the results of the homework assignment. Due date
Introduction to Machine Learning CMU-10701
Introduction to Machine Learning CMU-10701 23. Decision Trees Barnabás Póczos Contents Decision Trees: Definition + Motivation Algorithm for Learning Decision Trees Entropy, Mutual Information, Information
Decision Tree Analysis for Classification Problems. Entscheidungsunterstützungssysteme SS 18
Decision Tree Analysis for Classification Problems Entscheidungsunterstützungssysteme SS 18 Supervised segmentation An intuitive way of thinking about extracting patterns from data in a supervised manner
Problem. Problem Given a dictionary and a word. Which page (if any) contains the given word? 3 / 26
Binary Search Introduction Problem Problem Given a dictionary and a word. Which page (if any) contains the given word? 3 / 26 Strategy 1: Random Search Randomly select a page until the page containing
Decision trees. Decision tree induction - Algorithm ID3
Decision trees A decision tree is a predictive model which maps observations about an item to conclusions about the item's target value. Another name for such tree models is classification trees. In these
CS145: INTRODUCTION TO DATA MINING
CS145: INTRODUCTION TO DATA MINING 4: Vector Data: Decision Tree Instructor: Yizhou Sun yzsun@cs.ucla.edu October 10, 2017 Methods to Learn Vector Data Set Data Sequence Data Text Data Classification Clustering
Machine Learning & Data Mining
Group M L D Machine Learning M & Data Mining Chapter 7 Decision Trees Xin-Shun Xu @ SDU School of Computer Science and Technology, Shandong University Top 10 Algorithm in DM #1: C4.5 #2: K-Means #3: SVM
Data Mining Classification
Data Mining Classification Jingpeng Li 1 of 27 What is Classification? Assigning an object to a certain class based on its similarity to previous examples of other objects Can be done with reference to
Data Mining. CS57300 Purdue University. Bruno Ribeiro. February 8, 2018
Data Mining CS57300 Purdue University Bruno Ribeiro February 8, 2018 Decision trees Why Trees? interpretable/intuitive, popular in medical applications because they mimic the way a doctor thinks model
Informal Definition: Telling things apart
9. Decision Trees Informal Definition: Telling things apart 2 Nominal data No numeric feature vector Just a list or properties: Banana: longish, yellow Apple: round, medium sized, different colors like
Predictive Modeling: Classification. KSE 521 Topic 6 Mun Yi
Predictive Modeling: Classification Topic 6 Mun Yi Agenda Models and Induction Entropy and Information Gain Tree-Based Classifier Probability Estimation 2 Introduction Key concept of BI: Predictive modeling
Jeffrey D. Ullman Stanford University
Jeffrey D. Ullman Stanford University 3 We are given a set of training examples, consisting of input-output pairs (x,y), where: 1. x is an item of the type we want to evaluate. 2. y is the value of some
Induction of Decision Trees
Induction of Decision Trees Peter Waiganjo Wagacha This notes are for ICS320 Foundations of Learning and Adaptive Systems Institute of Computer Science University of Nairobi PO Box 30197, 00200 Nairobi.
CSE-4412(M) Midterm. There are five major questions, each worth 10 points, for a total of 50 points. Points for each sub-question are as indicated.
22 February 2007 CSE-4412(M) Midterm p. 1 of 12 CSE-4412(M) Midterm Sur / Last Name: Given / First Name: Student ID: Instructor: Parke Godfrey Exam Duration: 75 minutes Term: Winter 2007 Answer the following
CS6375: Machine Learning Gautam Kunapuli. Decision Trees
Gautam Kunapuli Example: Restaurant Recommendation Example: Develop a model to recommend restaurants to users depending on their past dining experiences. Here, the features are cost (x ) and the user s
Classification: Decision Trees
Classification: Decision Trees Outline Top-Down Decision Tree Construction Choosing the Splitting Attribute Information Gain and Gain Ratio 2 DECISION TREE An internal node is a test on an attribute. A
Examination Artificial Intelligence Module Intelligent Interaction Design December 2014
Examination Artificial Intelligence Module Intelligent Interaction Design December 2014 Introduction This exam is closed book, you may only use a simple calculator (addition, substraction, multiplication
Decision Tree Learning Lecture 2
Machine Learning Coms-4771 Decision Tree Learning Lecture 2 January 28, 2008 Two Types of Supervised Learning Problems (recap) Feature (input) space X, label (output) space Y. Unknown distribution D over
Error Error Δ = 0.44 (4/9)(0.25) (5/9)(0.2)
Chapter 4, Problem 3: For the classification function, the number of + classifications are 4 and the number of classifications are 5. Hence, the entropy of this collection with respect to the + class:
Introduction to Information Theory and Its Applications
Introduction to Information Theory and Its Applications Radim Bělohlávek Information Theory: What and Why information: one of key terms in our society: popular keywords such as information/knowledge society
Data Mining Classification: Basic Concepts, Decision Trees, and Model Evaluation
Data Mining Classification: Basic Concepts, Decision Trees, and Model Evaluation Lecture Notes for Chapter 4 Part I Introduction to Data Mining by Tan, Steinbach, Kumar Adapted by Qiang Yang (2010) Tan,Steinbach,
Decision Tree And Random Forest
Decision Tree And Random Forest Dr. Ammar Mohammed Associate Professor of Computer Science ISSR, Cairo University PhD of CS ( Uni. Koblenz-Landau, Germany) Spring 2019 Contact: mailto: Ammar@cu.edu.eg
Decision Trees Part 1. Rao Vemuri University of California, Davis
Decision Trees Part 1 Rao Vemuri University of California, Davis Overview What is a Decision Tree Sample Decision Trees How to Construct a Decision Tree Problems with Decision Trees Classification Vs Regression
Learning Decision Trees
Learning Decision Trees CS194-10 Fall 2011 Lecture 8 CS194-10 Fall 2011 Lecture 8 1 Outline Decision tree models Tree construction Tree pruning Continuous input features CS194-10 Fall 2011 Lecture 8 2
CS 6375 Machine Learning
CS 6375 Machine Learning Decision Trees Instructor: Yang Liu 1 Supervised Classifier X 1 X 2. X M Ref class label 2 1 Three variables: Attribute 1: Hair = {blond, dark} Attribute 2: Height = {tall, short}
Machine Learning: Exercise Sheet 2
Machine Learning: Exercise Sheet 2 Manuel Blum AG Maschinelles Lernen und Natürlichsprachliche Systeme Albert-Ludwigs-Universität Freiburg mblum@informatik.uni-freiburg.de Manuel Blum Machine Learning
Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen. Intelligent Data Analysis. Decision Trees
Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen Intelligent Data Analysis Decision Trees Paul Prasse, Niels Landwehr, Tobias Scheffer Decision Trees One of many applications:
Knowledge Discovery and Data Mining
Knowledge Discovery and Data Mining Lecture 06 - Regression & Decision Trees Tom Kelsey School of Computer Science University of St Andrews http://tom.home.cs.st-andrews.ac.uk twk@st-andrews.ac.uk Tom
Introduction to Information Theory and Its Applications. Dept. Computer Science Palacký University, Olomouc
Introduction to Information Theory and Its Applications Radim Bělohlávek Dept. Computer Science Palacký University, Olomouc radim.belohlavek@acm.org Outline This is a preliminary version of a text providing
Classification and Prediction
Classification Classification and Prediction Classification: predict categorical class labels Build a model for a set of classes/concepts Classify loan applications (approve/decline) Prediction: model
Machine Learning 2nd Edi7on
Lecture Slides for INTRODUCTION TO Machine Learning 2nd Edi7on CHAPTER 9: Decision Trees ETHEM ALPAYDIN The MIT Press, 2010 Edited and expanded for CS 4641 by Chris Simpkins alpaydin@boun.edu.tr h1p://www.cmpe.boun.edu.tr/~ethem/i2ml2e
Information theory and decision tree
Information theory and decision tree Jianxin Wu LAMDA Group National Key Lab for Novel Software Technology Nanjing University, China wujx2001@gmail.com June 14, 2018 Contents 1 Prefix code and Huffman
Learning Decision Trees
Learning Decision Trees Machine Learning Fall 2018 Some slides from Tom Mitchell, Dan Roth and others 1 Key issues in machine learning Modeling How to formulate your problem as a machine learning problem?
Decision Tree Learning
Decision Tree Learning Berlin Chen Department of Computer Science & Information Engineering National Taiwan Normal University References: 1. Machine Learning, Chapter 3 2. Data Mining: Concepts, Models,
Decision Trees / NLP Introduction
Decision Trees / NLP Introduction Dr. Kevin Koidl School of Computer Science and Statistic Trinity College Dublin ADAPT Research Centre The ADAPT Centre is funded under the SFI Research Centres Programme
Decision Trees. CS 341 Lectures 8/9 Dan Sheldon
Decision rees CS 341 Lectures 8/9 Dan Sheldon Review: Linear Methods Y! So far, we ve looked at linear methods! Linear regression! Fit a line/plane/hyperplane X 2 X 1! Logistic regression! Decision boundary
Einführung in Web- und Data-Science
Einführung in Web- und Data-Science Prof. Dr. Ralf Möller Universität zu Lübeck Institut für Informationssysteme Tanya Braun (Übungen) Inductive Learning Chapter 18/19 Chapters 3 and 4 Material adopted
Discrete Multivariate Statistics
Discrete Multivariate Statistics Univariate Discrete Random variables Let X be a discrete random variable which, in this module, will be assumed to take a finite number of t different values which are
http://xkcd.com/1570/ Strategy: Top Down Recursive divide-and-conquer fashion First: Select attribute for root node Create branch for each possible attribute value Then: Split
Bayes Decision Theory
Bayes Decision Theory Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjin@postech.ac.kr 1 / 16
Classification Using Decision Trees
Classification Using Decision Trees 1. Introduction Data mining term is mainly used for the specific set of six activities namely Classification, Estimation, Prediction, Affinity grouping or Association
Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen. Decision Trees. Tobias Scheffer
Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen Decision Trees Tobias Scheffer Decision Trees One of many applications: credit risk Employed longer than 3 months Positive credit
Lecture 24: Other (Non-linear) Classifiers: Decision Tree Learning, Boosting, and Support Vector Classification Instructor: Prof. Ganesh Ramakrishnan
Lecture 24: Other (Non-linear) Classifiers: Decision Tree Learning, Boosting, and Support Vector Classification Instructor: Prof Ganesh Ramakrishnan October 20, 2016 1 / 25 Decision Trees: Cascade of step
Rule Generation using Decision Trees
Rule Generation using Decision Trees Dr. Rajni Jain 1. Introduction A DT is a classification scheme which generates a tree and a set of rules, representing the model of different classes, from a given
Lecture 17: Trees and Merge Sort 10:00 AM, Oct 15, 2018
CS17 Integrated Introduction to Computer Science Klein Contents Lecture 17: Trees and Merge Sort 10:00 AM, Oct 15, 2018 1 Tree definitions 1 2 Analysis of mergesort using a binary tree 1 3 Analysis of
EVALUATING RISK FACTORS OF BEING OBESE, BY USING ID3 ALGORITHM IN WEKA SOFTWARE
EVALUATING RISK FACTORS OF BEING OBESE, BY USING ID3 ALGORITHM IN WEKA SOFTWARE Msc. Daniela Qendraj (Halidini) Msc. Evgjeni Xhafaj Department of Mathematics, Faculty of Information Technology, University
CLUe Training An Introduction to Machine Learning in R with an example from handwritten digit recognition
CLUe Training An Introduction to Machine Learning in R with an example from handwritten digit recognition Ad Feelders Universiteit Utrecht Department of Information and Computing Sciences Algorithmic Data
Asymptotic redundancy and prolixity
Asymptotic redundancy and prolixity Yuval Dagan, Yuval Filmus, and Shay Moran April 6, 2017 Abstract Gallager (1978) considered the worst-case redundancy of Huffman codes as the maximum probability tends
Machine Learning and Data Mining. Decision Trees. Prof. Alexander Ihler
+ Machine Learning and Data Mining Decision Trees Prof. Alexander Ihler Decision trees Func-onal form f(x;µ): nested if-then-else statements Discrete features: fully expressive (any func-on) Structure:
Decision Trees. Common applications: Health diagnosis systems Bank credit analysis
Decision Trees Rodrigo Fernandes de Mello Invited Professor at Télécom ParisTech Associate Professor at Universidade de São Paulo, ICMC, Brazil http://www.icmc.usp.br/~mello mello@icmc.usp.br Decision
2018 CS420, Machine Learning, Lecture 5. Tree Models. Weinan Zhang Shanghai Jiao Tong University
2018 CS420, Machine Learning, Lecture 5 Tree Models Weinan Zhang Shanghai Jiao Tong University http://wnzhang.net http://wnzhang.net/teaching/cs420/index.html ML Task: Function Approximation Problem setting
Learning Decision Trees
Learning Decision Trees Machine Learning Spring 2018 1 This lecture: Learning Decision Trees 1. Representation: What are decision trees? 2. Algorithm: Learning decision trees The ID3 algorithm: A greedy
Administrative notes. Computational Thinking ct.cs.ubc.ca
Administrative notes Labs this week: project time. Remember, you need to pass the project in order to pass the course! (See course syllabus.) Clicker grades should be on-line now Administrative notes March
Decision Tree Learning
Decision Tree Learning Goals for the lecture you should understand the following concepts the decision tree representation the standard top-down approach to learning a tree Occam s razor entropy and information
Inductive Learning. Chapter 18. Material adopted from Yun Peng, Chuck Dyer, Gregory Piatetsky-Shapiro & Gary Parker
Inductive Learning Chapter 18 Material adopted from Yun Peng, Chuck Dyer, Gregory Piatetsky-Shapiro & Gary Parker Chapters 3 and 4 Inductive Learning Framework Induce a conclusion from the examples Raw
Final Exam, Machine Learning, Spring 2009
Name: Andrew ID: Final Exam, 10701 Machine Learning, Spring 2009 - The exam is open-book, open-notes, no electronics other than calculators. - The maximum possible score on this exam is 100. You have 3
CS 380: ARTIFICIAL INTELLIGENCE MACHINE LEARNING. Santiago Ontañón
CS 380: ARTIFICIAL INTELLIGENCE MACHINE LEARNING Santiago Ontañón so367@drexel.edu Summary so far: Rational Agents Problem Solving Systematic Search: Uninformed Informed Local Search Adversarial Search
N= {1,2,3,4,5,6,7,8,9,10,11,...}
1.1: Integers and Order of Operations 1. Define the integers 2. Graph integers on a number line. 3. Using inequality symbols < and > 4. Find the absolute value of an integer 5. Perform operations with
( D) I(2,3) I(4,0) I(3,2) weighted avg. of entropies
Decision Tree Induction using Information Gain Let I(x,y) as the entropy in a dataset with x number of class 1(i.e., play ) and y number of class (i.e., don t play outcomes. The entropy at the root, i.e.,
Lecture 3: Decision Trees
Lecture 3: Decision Trees Cognitive Systems II - Machine Learning SS 2005 Part I: Basic Approaches of Concept Learning ID3, Information Gain, Overfitting, Pruning Lecture 3: Decision Trees p. Decision
Decision Trees Entropy, Information Gain, Gain Ratio
Changelog: 14 Oct, 30 Oct Decision Trees Entropy, Information Gain, Gain Ratio Lecture 3: Part 2 Outline Entropy Information gain Gain ratio Marina Santini Acknowledgements Slides borrowed and adapted
A Decision Stump. Decision Trees, cont. Boosting. Machine Learning 10701/15781 Carlos Guestrin Carnegie Mellon University. October 1 st, 2007
Decision Trees, cont. Boosting Machine Learning 10701/15781 Carlos Guestrin Carnegie Mellon University October 1 st, 2007 1 A Decision Stump 2 1 The final tree 3 Basic Decision Tree Building Summarized
Generalization to Multi-Class and Continuous Responses. STA Data Mining I
Generalization to Multi-Class and Continuous Responses STA 5703 - Data Mining I 1. Categorical Responses (a) Splitting Criterion Outline Goodness-of-split Criterion Chi-square Tests and Twoing Rule (b)
Decision Tree Learning and Inductive Inference
Decision Tree Learning and Inductive Inference 1 Widely used method for inductive inference Inductive Inference Hypothesis: Any hypothesis found to approximate the target function well over a sufficiently
Machine Learning Alternatives to Manual Knowledge Acquisition
Machine Learning Alternatives to Manual Knowledge Acquisition Interactive programs which elicit knowledge from the expert during the course of a conversation at the terminal. Programs which learn by scanning
Finding the Gold in Your Data
Finding the Gold in Your Data An introduction to Data Mining Originally presented @ SAS Global Forum David A. Dickey North Carolina State University Decision Trees A divisive method (splits) Start with
Final Exam. 1 True or False (15 Points)
10-606 Final Exam Submit by Oct. 16, 2017 11:59pm EST Please submit early, and update your submission if you want to make changes. Do not wait to the last minute to submit: we reserve the right not to
Decision Trees.
. Machine Learning Decision Trees Prof. Dr. Martin Riedmiller AG Maschinelles Lernen und Natürlichsprachliche Systeme Institut für Informatik Technische Fakultät Albert-Ludwigs-Universität Freiburg riedmiller@informatik.uni-freiburg.de
Classification and Regression Trees
Classification and Regression Trees Ryan P Adams So far, we have primarily examined linear classifiers and regressors, and considered several different ways to train them When we ve found the linearity
Outline. Computer Science 331. Cost of Binary Search Tree Operations. Bounds on Height: Worst- and Average-Case
Outline Computer Science Average Case Analysis: Binary Search Trees Mike Jacobson Department of Computer Science University of Calgary Lecture #7 Motivation and Objective Definition 4 Mike Jacobson (University
Decision trees COMS 4771
Decision trees COMS 4771 1. Prediction functions (again) Learning prediction functions IID model for supervised learning: (X 1, Y 1),..., (X n, Y n), (X, Y ) are iid random pairs (i.e., labeled examples).
Machine Learning 2010
Machine Learning 2010 Decision Trees Email: mrichter@ucalgary.ca -- 1 - Part 1 General -- 2 - Representation with Decision Trees (1) Examples are attribute-value vectors Representation of concepts by labeled
The Solution to Assignment 6
The Solution to Assignment 6 Problem 1: Use the 2-fold cross-validation to evaluate the Decision Tree Model for trees up to 2 levels deep (that is, the maximum path length from the root to the leaves is
Machine Learning 3. week
Machine Learning 3. week Entropy Decision Trees ID3 C4.5 Classification and Regression Trees (CART) 1 What is Decision Tree As a short description, decision tree is a data classification procedure which
Decision trees for stream data mining new results
Decision trees for stream data mining new results Leszek Rutkowski leszek.rutkowski@iisi.pcz.pl Lena Pietruczuk lena.pietruczuk@iisi.pcz.pl Maciej Jaworski maciej.jaworski@iisi.pcz.pl Piotr Duda piotr.duda@iisi.pcz.pl
Decision Trees. Introduction. Some facts about decision trees: They represent data-classification models.
Decision Trees Introduction Some facts about decision trees: They represent data-classification models. An internal node of the tree represents a question about feature-vector attribute, and whose answer
Information Theory with Applications, Math6397 Lecture Notes from September 30, 2014 taken by Ilknur Telkes
Information Theory with Applications, Math6397 Lecture Notes from September 3, 24 taken by Ilknur Telkes Last Time Kraft inequality (sep.or) prefix code Shannon Fano code Bound for average code-word length
Information Theory, Statistics, and Decision Trees
Information Theory, Statistics, and Decision Trees Léon Bottou COS 424 4/6/2010 Summary 1. Basic information theory. 2. Decision trees. 3. Information theory and statistics. Léon Bottou 2/31 COS 424 4/6/2010
DECISION TREE LEARNING. [read Chapter 3] [recommended exercises 3.1, 3.4]
![DECISION TREE LEARNING. [read Chapter 3] [recommended exercises 3.1, 3.4]](/thumbs/78/78784288.jpg "DECISION TREE LEARNING. [read Chapter 3] [recommended exercises 3.1, 3.4]")
1 DECISION TREE LEARNING [read Chapter 3] [recommended exercises 3.1, 3.4] Decision tree representation ID3 learning algorithm Entropy, Information gain Overfitting Decision Tree 2 Representation: Tree-structured
Apprentissage automatique et fouille de données (part 2)
Apprentissage automatique et fouille de données (part 2) Telecom Saint-Etienne Elisa Fromont (basé sur les cours d Hendrik Blockeel et de Tom Mitchell) 1 Induction of decision trees : outline (adapted
Decision Trees. Lewis Fishgold. (Material in these slides adapted from Ray Mooney's slides on Decision Trees)
Decision Trees Lewis Fishgold (Material in these slides adapted from Ray Mooney's slides on Decision Trees) Classification using Decision Trees Nodes test features, there is one branch for each value of
15-381: Artificial Intelligence. Decision trees
15-381: Artificial Intelligence Decision trees Bayes classifiers find the label that maximizes: Naïve Bayes models assume independence of the features given the label leading to the following over documents
Decision Trees.
. Machine Learning Decision Trees Prof. Dr. Martin Riedmiller AG Maschinelles Lernen und Natürlichsprachliche Systeme Institut für Informatik Technische Fakultät Albert-Ludwigs-Universität Freiburg riedmiller@informatik.uni-freiburg.de
Midterm, Fall 2003
5-78 Midterm, Fall 2003 YOUR ANDREW USERID IN CAPITAL LETTERS: YOUR NAME: There are 9 questions. The ninth may be more time-consuming and is worth only three points, so do not attempt 9 unless you are
Algorithms for Classification: The Basic Methods
Algorithms for Classification: The Basic Methods Outline Simplicity first: 1R Naïve Bayes 2 Classification Task: Given a set of pre-classified examples, build a model or classifier to classify new cases.
STAT Chapter 13: Categorical Data. Recall we have studied binomial data, in which each trial falls into one of 2 categories (success/failure).
STAT 515 -- Chapter 13: Categorical Data Recall we have studied binomial data, in which each trial falls into one of 2 categories (success/failure). Many studies allow for more than 2 categories. Example
Lecture 28 Chi-Square Analysis
Lecture 28 STAT 225 Introduction to Probability Models April 23, 2014 Whitney Huang Purdue University 28.1 χ 2 test for For a given contingency table, we want to test if two have a relationship or not
Chapter 6: Classification
Chapter 6: Classification 1) Introduction Classification problem, evaluation of classifiers, prediction 2) Bayesian Classifiers Bayes classifier, naive Bayes classifier, applications 3) Linear discriminant
Text Mining. Dr. Yanjun Li. Associate Professor. Department of Computer and Information Sciences Fordham University
Text Mining Dr. Yanjun Li Associate Professor Department of Computer and Information Sciences Fordham University Outline Introduction: Data Mining Part One: Text Mining Part Two: Preprocessing Text Data
PART III. Outline. Codes and Cryptography. Sources. Optimal Codes (I) Jorge L. Villar. MAMME, Fall 2015
Outline Codes and Cryptography 1 Information Sources and Optimal Codes 2 Building Optimal Codes: Huffman Codes MAMME, Fall 2015 3 Shannon Entropy and Mutual Information PART III Sources Information source: