Artificial Intelligence EDAF70
|
|
|
- Adam Chambers
- 6 years ago
- Views:
Transcription
1 Artificial Intelligence EDAF70 Pierre Nugues Lund University February 7, 2018 Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
2 Why Machine Learning: Early Artificial Intelligence Early artificial intelligence techniques used introspection to codify knowledge, often in the form of rules. Expert systems, one of the most notable applications of traditional AI, were entirely based on the competence of experts. This has two major drawbacks: Need of an expertise to understand and explain the rules Bias introduced by the expert Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
3 Why Machine Learning: What has Changed Now terabytes of data available. Makes it impossible to understand such volumes of data, organize them using manually-crafted rules. Triggered a major move to empirical and statistical techniques. In fact, most machine learning techniques come from traditional statistics. Applications in natural language processing, medicine, banking, online shopping, image recognition, etc. The success of companies like Google, Facebook, Amazon, and Netflix, not to mention Wall Street firms and industries from manufacturing and retail to healthcare, is increasingly driven by better tools for extracting meaning from very large quantities of data. Data Scientist is now the hottest job title in Silicon Valley. Tim O Reilly Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
4 Some Definitions 1 Machine learning always starts with data sets: a collection of objects or observations. 2 Machine-learning algorithms can be classified along two main lines: supervised and unsupervised classification. 3 Supervised algorithms need a training set, where the objects are described in terms of attributes and belong to a known class or have a known output. 4 The performance of the resulting classifier is measured against a test set. 5 We can also use N-fold cross validation, where the test set is selected randomly from the training set N times, usually Unsupervised algorithms consider objects, where no class is provided. 7 Unsupervised algorithms learn regularities in data sets. Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
5 A Data Set: Salammbô A corpus is a collection a body of texts. French original English translation Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
6 Supervised Learning Letter counts from Salammbô Chapter French English # characters # A # characters # A Chapter 1 36,961 2,503 35,680 2,217 Chapter 2 43,621 2,992 42,514 2,761 Chapter 3 15,694 1,042 15, Chapter 4 36,231 2,487 35,298 2,274 Chapter 5 29,945 2,014 29,800 1,865 Chapter 6 40,588 2,805 40,255 2,606 Chapter 7 75,255 5,062 74,532 4,805 Chapter 8 37,709 2,643 37,464 2,396 Chapter 9 30,899 2,126 31,030 1,993 Chapter 10 25,486 1,784 24,843 1,627 Chapter 11 37,497 2,641 36,172 2,375 Chapter 12 40,398 2,766 39,552 2,560 Chapter 13 74,105 5,047 72,545 4,597 Chapter 14 76,725 5,312 75,352 4,871 Chapter 15 18,317 1,215 18,031 1,119 Data set: programs/ch04/salammbo Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
7 Supervised Learning: Regression Letter count from Salammbô in French "salammbo_a_fr.tsv" f(x) Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
8 Models We will assume that data sets are governed by functions or models. For instance given the set: there exists a function such that: {(x i,y i ) 0 < i N}, f (x i ) = y i. Supervised machine learning algorithms will produce hypothesized functions or models fitting the data. Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
9 Selecting a Model Often, multiple models can fit a data set: Three polynomials of degree: 1, 8, and 9 A general rule in machine learning is to prefer the simplest hypotheses, here the lower polynomial degrees. Source code: Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
10 Regression: Another Model, the Logistic Curve (Verhulst) Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
11 Supervised Learning: Classification A data set for binary classification. # char. # A class (y) # char. # A class (y) Chapter 1 36,961 2, ,680 2,217 0 Chapter 2 43,621 2, ,514 2,761 0 Chapter 3 15,694 1, , Chapter 4 36,231 2, ,298 2,274 0 Chapter 5 29,945 2, ,800 1,865 0 Chapter 6 40,588 2, ,255 2,606 0 Chapter 7 75,255 5, ,532 4,805 0 Chapter 8 37,709 2, ,464 2,396 0 Chapter 9 30,899 2, ,030 1,993 0 Chapter 10 25,486 1, ,843 1,627 0 Chapter 11 37,497 2, ,172 2,375 0 Chapter 12 40,398 2, ,552 2,560 0 Chapter 13 74,105 5, ,545 4,597 0 Chapter 14 76,725 5, ,352 4,871 0 Chapter 15 18,317 1, ,031 1,119 0 Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
12 Supervised Learning: Regression and Classification Given the data set, {(x i,y i ) 0 < i N} and a model f : Classification: f (x) = y is discrete, Regression: f (x) = y is continuous. Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
13 Types of Iris Iris virginica Iris setosa Iris versicolor Courtesy Wikipedia Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
14 Supervised Learning: Fisher s Iris data set (1936) Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
15 Unsupervised Learning: Clustering No class is given: Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
16 Objects, Attributes, and Classes. After Quinlan (1986) Object Attributes Class Outlook Temperature Humidity Windy 1 Sunny Hot High False N 2 Sunny Hot High True N 3 Overcast Hot High False P 4 Rain Mild High False P 5 Rain Cool Normal False P 6 Rain Cool Normal True N 7 Overcast Cool Normal True P 8 Sunny Mild High False N 9 Sunny Cool Normal False P 10 Rain Mild Normal False P 11 Sunny Mild Normal True P 12 Overcast Mild High True P 13 Overcast Hot Normal False P 14 Rain Mild High True N Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
17 Matrix Notation A feature vector (predictors or attributes): x, and feature matrix: X; The class: y and the class vector: y; The predicted class (response): ŷ, and predicted class vector: ŷ X = Sunny Hot High False Sunny Hot High True Overcast Hot High False Rain Mild High False Rain Cool Normal False Rain Cool Normal True Overcast Cool Normal True Sunny Mild High False Sunny Cool Normal False Rain Mild Normal False Sunny Mild Normal True Overcast Mild High True Overcast Hot Normal False Rain Mild High True ;y = N N P P P N P N P P P P P N Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
18 Classifying Objects with Decision Trees. After Quinlan (1986) Outlook P: 9, N: 5 Humidity P: 2, N: 3 sunny overcast P: 4 rain Windy P: 3, N: 2 high normal true false N: 3 P: 2 N: 2 P: 3 Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
19 Decision Trees and Classification Each object is defined by an attribute vector (or feature vector or predictors) {A 1,A 2,...,A v } Each object belongs to one class {C 1,C 2,...,C n } The attributes of the examples are: {Outlook, Temperature, Humidity, Windy} and the classes are: {N,P}. The nodes of the tree are the attributes. Each attribute has a set of possible values. The values of Outlook are {sunny, rain, overcast} The branches correspond to the values of each attribute The optimal tree corresponds to a minimal number of tests. Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
20 Entropy, Decision Trees, and Classification Decision trees are useful devices to classify objects into a set of classes. Shannon s entropy is a quantification of distribution diversity in a set of symbols It can help us learn automatically decision trees from a set of data. The algorithm is one of the simplest machine-learning techniques to obtain a classifier. Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
21 Inducing (Learning) Decision Trees Automatically: ID3 It is possible to design many trees that classify the objects successfully An efficient decision tree uses a minimal number of tests. In the decision tree: Each example is defined by a finite number of attributes. Each node in the decision tree corresponds to an attribute that has as many branches as the attribute has possible values. At the root of the tree, the condition must be the most discriminating, that is, have branches gathering most positive examples while others gather negative examples. Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
22 ID3 (Quinlan 1986) Each attribute scatters the set into as many subsets as there are values for this attribute. In the data set, we have nine positive examples and five negative ones, Outlook P: 9, N: 5 sunny overcast rain P: 2, N: 3 P: 4 P: 3, N: 2 At each decision point, the best attribute has the maximal separation power, the maximal information gain Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
23 Better Root? hot Temperature P: 9, N: 5 mild cool P: 2, N: 2 P: 4, N: 2 P: 3, N: 1 How can we measure this? Answer: Entropy or Gini impurity Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
24 Entropy Information theory models a text as a sequence of symbols. Let x 1,x 2,...,x N be a discrete set of N symbols representing the characters. The information content of a symbol is defined as where I (x i ) = log 2 P(x i ) = log 2 1 P(x i ), P(x i ) = Count(x i) n j=1 Count(x j). Entropy, the average information content, is defined as: By convention: 0log 2 0 = 0. H(X ) = P(x)log 2 P(x), x X Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
25 Entropy of the Dataset sunny Outlook P: 9, N: 5 overcast rain P: 2, N: 3 P: 4 P: 3, N: 2 Entropy of the top node: In the three child nodes: I (p,n) = 9 14 log 2 sunny : I (p 1,n 1 ) = 2 5 log 2 overcast : I (p 2,n 2 ) = 0.0 rain : I (p 3,n 3 ) = 3 5 log log = log = log = Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
26 Understanding the Entropy For a two-class set, we set: x = p p + n and n p + n = 1 x. I (x) = x log 2 x (1 x)log 2 (1 x) with x [0,1]. The entropy reaches a maximum when there are as many positive as negative examples in the data set. It is minimal when the set consists of either positive or negative examples. Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
27 Gini Impurity Gini impurity is an alternative to entropy: For a two-class set, we set: G(X ) = P(x)(1 P(x)), x X 2G(x) = 2(x(1 x) + (1 x)x) with x [0,1]. = 4x(1 x) Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
28 Entropy of a Text The entropy of the text is H(X ) = x X P(x)log 2 P(x). = P(A)log 2 P(A) P(B)log 2 P(B)... P(Z)log 2 P(Z) P(À)log 2 P(À)... P(Ÿ )log 2 P(Ÿ ) P(blanks)log 2 P(blanks). Entropy of Gustave Flaubert s Salammbô in French is H(X ) = Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
29 Cross-Entropy The cross entropy of M on P is defined as: H(P,M) = P(x)log 2 M(x). x X We have the inequality H(P) H(P,M). The difference is called the Kullback-Leibler divergence. Entropy Cross entropy Difference Salammbô, chapters 1-14, training set Salammbô, chapter 15, test set Notre Dame de Paris, test set Nineteen Eighty-Four, test set Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
30 ID3 (Quinlan 1986) ID3 uses the entropy to select the best attribute to be the root of the tree and recursively the next attributes of the resulting nodes. The entropy of a two-class set p and n is: I (p,n) = p p + n log 2 p p + n n p + n log 2 n p + n. The weighted average of all the nodes below an attribute A is: E(A) = v i=1 p i + n i p + n I ( p i n i, ). p i + n i p i + n i The information gain is defined as I (p,n) E(A) (or I before I after ) This measures the separating power of an attribute: the more the gain, the better the attribute. Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
31 Example The entropy of the data set is: I (p,n) = 9 14 log log = Outlook has three values: sunny, overcast, and rain. The entropies of the corresponding subsets are: sunny : I (p 1,n 1 ) = 2 5 log 2 overcast : I (p 2,n 2 ) = 0.0 rain : I (p 3,n 3 ) = 3 5 log log = log = Gain(Outlook) is then = 0.246, the 14 highest possible among the attributes. We have Gain(Temperature) = 0.029, Gain(Humidity) = 0.151, and Gain(Windy) = Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
32 Inducing Decision Trees Automatically: ID3 The algorithm to build the decision tree is simple. The information gain is computed on the data set for all attributes in the set of attributes, SA. The attribute with the highest gain is selected to be the root of the tree: A = arg maxi (n,p) E(a). a SA The data set is split into v subsets {N 1,...,N v }, where the value of A for the objects in N i is A i For each subset, a corresponding node is created below the root. This process is repeated recursively for each node of the tree with the subset it contains until all the objects of the node are either positive or negative. For a training set of N instances each having M attributes, ID3 s complexity to generate a decision tree is O(NM). Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
33 Decision Tree Learning Algorithm (From the Textbook) 1: function DTL(examples, attributes, parent_examples) returns a tree 2: if examples is empty then 3: return PluralityValue(parent_examples) 4: else if all examples have the same classification then 5: return the classification 6: else if attributes is empty then 7: return PluralityValue(examples) 8: else 9: A argmax a attributes InformationGain(a,examples) 10: tree a new decision tree with root test A 11: for all v k A do 12: exs {e : e examples and e.a = v k } 13: subtree DTL(exs, attributes A, examples) 14: add a branch to tree with label {A = v k } and subtree subtree return tree Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
34 How do we know that h f? (Hume s Problem of Induction) Learning Curve 1) Use theorems of computational/statistical learning theory The classical evaluation technique uses a training set and a test set. 2) Generally, Try h onthe a new larger test theset training of examples set, the better the performance. This (usecan same be visualized distribution with a learning over example curve. Fromspace the textbook, as training Stuart set) Russell and Peter Norvig, Artificial Intelligence, 3rd ed., 2010, page 703. Learning curve = % correct on test set as a function of training set size 1 % correct on test set Training set size Pierre Nugues Artificial Intelligence EDAF70 February Chapter 7, , Sections / 54 2
35 Overfitting When two classifiers have equal performances on a specific test set, the simplest one is supposed to be more general A small decision tree is always preferable to a larger one. Complex classifiers may show an overfit to the training data and have poor performance when the data set changes. In the case of decision trees, overfits show with deep trees and when numerous leaves have few examples. Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
36 Pruning Trees Decision tree pruning consists in removing and merging leaves with few objects, for instance one or two. To prune the tree: 1 Generate the complete decision tree 2 Consider the nodes that have only leaves as children and merge the leaves if the information gain is below a certain threshold. 3 To determine the threshold, we use a significance test. This is the χ 2 pruning. Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
37 Contingency Tables Example: A set of 12 positive and 8 negative examples. Let us compare two attributes: Attribute 1 with three values:{v 1 1,v2 1,v3 1 } Attribute 2 with three values:{v 1 2,v2 2,v3 2 } Contingency tables are devices to visualize frequency distributions. Attribute 1 v1 1 v1 2 v1 3 Positive (60%) Negative (40%) Total (100%) Attribute 2 v2 1 v2 2 v2 3 Positive (60%) Negative (40%) Total (100%) Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
38 Significance Test An attribute with no significance would have the same proportions of positive and negative examples before and after the test. Proportions before, for instance 12 and 8: p p + n and n p + n Proportions after, for v k, a value of the attribute: p k p k + n k and n k p k + n k The expected values of a nonsignificant attribute (null hypothesis) are: ˆp k = p p + n (p k + n k ) ˆn k = n p + n (p k + n k ) Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
39 Example Contingency tables with the expected frequencies, in red. Attribute 1 v1 1 v1 1 v1 2 v1 2 v1 3 v1 3 Positive (60%) Negative (40%) Total (100%) Attribute 2 v1 1 v1 1 v1 2 v1 2 v1 3 v1 3 Positive (60%) Negative (40%) Total (100%) The total deviation for an attribute with values v 1..v d is: = d k=1 (p k ˆp k ) 2 ˆp k + (n k ˆn k ) 2 ˆn k Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
40 Pruning Trees (II) The null hypothesis corresponds to an irrelevant attribute: = 0. values are distributed according to the χ 2 distribution If v is the number of values of an attribute, the degree of freedom is v 1 The corresponding χ 2 distributions can be found in statistical tables. For three degrees of freedom: 10%: 6.25, 5%: 7.81, 1%: 11.34, 1 : The significance level is usually 5% An alternate pruning method is to modify the termination condition of the algorithm: We stop when the information gain is below a certain threshold. Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
41 Dealing with Real Data Unknown attribute values: Use the most frequent value of the attribute or all the values; Attributes with many values (large attribute domains): Difficult to handle for decision trees. The book suggests to use the gain ratio: v Information gain Information value,where Information value = p i + n i i=1 p + n log p i + n i 2 p + n Numerical attributes: Find the binary split point that maximizes the information gain Numerical output: Regression tree Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
42 Evaluation The standard evaluation procedure is to train the classifier on a training set and evaluate the performance on a test set. When we have only one set, we divide it in two subsets: the training set and the test set (or holdout data). The split can be or This often optimizes the classifier for a specific test set and creates an overfit Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
43 Cross Validation A N-fold cross validation mitigates the overfit The set is partitioned into N subsets, N = 5 for example, one of them being the test set (red) and the rest the training set (gray). The process is repeated N times with a different test set: N folds fold 1 fold 2 fold 3 fold 4 fold 5 part 1 part 2 part 3 part 4 part 5 At the extreme, leave-one-out cross-validation Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
44 Model Selection Validation can apply to one classification method We can use it to select a classification method and its parametrization. Needs three sets: training set, development set, and test set. Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
45 Loss Depending on their type, errors can have different impacts. Imagine a smoke detector that predicts fire. Equivalent to a classifier that uses input data from sensors and classifies them as fire and nonfire. The loss function is defined as L(y,ŷ) with ŷ = h(x), where x is the vector of attributes, h the classifier, and y, the correct value. The cost of L(fire, nonfire) is much higher than L(nonfire, fire). L(fire,fire) = L(nonfire,nonfire) = 0 Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
46 Common Loss Measures L 1 (y,ŷ) = y ŷ Absolute value loss L 2 (y,ŷ) = (y ŷ) 2 Squared error loss L 0/1 (y,ŷ) = 0 if y = ŷ else 1 0/1 loss Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
47 Empirical Loss We compute the empirical loss of a classifier h on a set of examples E using the formula: Loss(L,E,h) = 1 N E L(y,h(x)). For continuous functions: Loss(L,E,h) = 1 N E (y h(x)) 2. Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
48 Evaluation There are different kinds of measures to evaluate the performance of machine learning techniques, for instance: Precision and recall in information retrieval and natural language processing; The receiver operating characteristic (ROC) in medicine. Positive examples: P Negative examples: N Classified as P True positives: A False positives: B Classified as N False negatives: C True negatives: D More on the receiver operating characteristic here: http: //en.wikipedia.org/wiki/receiver_operating_characteristic Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
49 Recall, Precision, and the F-Measure A D The accuracy is P N. Recall measures how much relevant examples the system has classified correctly, for P: Recall = A A C. Precision is the accuracy of what has been returned, for P: Precision = A A B. Recall and precision are combined into the F-measure, which is defined as the harmonic mean of both numbers: F = 2 Precision Recall. Precision + Recall Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
50 Measuring Quality: The Confusion Matrix A task in natural language processing: Identify the parts of speech (POS) of words. Example: The can rusted The human: The/art (DT) can/noun (NN) rusted/verb (VBD) The POS tagger: The/art (DT) can/modal (MD) rusted/verb (VBD) Correct Tagger DT IN JJ NN RB RP VB VBD VBG VBN DT IN JJ NN RB RP VB VBD VBG VBN After Franz (1996, p. 124) Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
51 The Weka Toolkit Weka: A powerful collection of machine-learning algorithms Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
52 The Weka Toolkit Running ID3 Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
53 ARFF: The Weka Data Format Storing Quinlan s data set in Weka s attribute-relation file format (ARFF) outlook {sunny, overcast, temperature {hot, mild, humidity {high, windy {TRUE, play {yes, sunny,hot,high,false,no sunny,hot,high,true,no overcast,hot,high,false,yes rainy,mild,high,false,yes rainy,cool,normal,false,yes rainy,cool,normal,true,no overcast,cool,normal,true,yes Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
54 Other Toolkits Scikit-learn: Keras: R: Matlab/Octave: H 2 O: Pierre Nugues Artificial Intelligence EDAF70 February 7, / 54
Classification: Decision Trees
Classification: Decision Trees Outline Top-Down Decision Tree Construction Choosing the Splitting Attribute Information Gain and Gain Ratio 2 DECISION TREE An internal node is a test on an attribute. A
Decision trees. Special Course in Computer and Information Science II. Adam Gyenge Helsinki University of Technology
Decision trees Special Course in Computer and Information Science II Adam Gyenge Helsinki University of Technology 6.2.2008 Introduction Outline: Definition of decision trees ID3 Pruning methods Bibliography:
CS 6375 Machine Learning
CS 6375 Machine Learning Decision Trees Instructor: Yang Liu 1 Supervised Classifier X 1 X 2. X M Ref class label 2 1 Three variables: Attribute 1: Hair = {blond, dark} Attribute 2: Height = {tall, short}
http://xkcd.com/1570/ Strategy: Top Down Recursive divide-and-conquer fashion First: Select attribute for root node Create branch for each possible attribute value Then: Split
The Solution to Assignment 6
The Solution to Assignment 6 Problem 1: Use the 2-fold cross-validation to evaluate the Decision Tree Model for trees up to 2 levels deep (that is, the maximum path length from the root to the leaves is
Learning Decision Trees
Learning Decision Trees Machine Learning Fall 2018 Some slides from Tom Mitchell, Dan Roth and others 1 Key issues in machine learning Modeling How to formulate your problem as a machine learning problem?
Decision Trees. Each internal node : an attribute Branch: Outcome of the test Leaf node or terminal node: class label.
Decision Trees Supervised approach Used for Classification (Categorical values) or regression (continuous values). The learning of decision trees is from class-labeled training tuples. Flowchart like structure.
Learning Decision Trees
Learning Decision Trees Machine Learning Spring 2018 1 This lecture: Learning Decision Trees 1. Representation: What are decision trees? 2. Algorithm: Learning decision trees The ID3 algorithm: A greedy
CS 380: ARTIFICIAL INTELLIGENCE MACHINE LEARNING. Santiago Ontañón
CS 380: ARTIFICIAL INTELLIGENCE MACHINE LEARNING Santiago Ontañón so367@drexel.edu Summary so far: Rational Agents Problem Solving Systematic Search: Uninformed Informed Local Search Adversarial Search
Decision Trees. Tirgul 5
Decision Trees Tirgul 5 Using Decision Trees It could be difficult to decide which pet is right for you. We ll find a nice algorithm to help us decide what to choose without having to think about it. 2
2018 CS420, Machine Learning, Lecture 5. Tree Models. Weinan Zhang Shanghai Jiao Tong University
2018 CS420, Machine Learning, Lecture 5 Tree Models Weinan Zhang Shanghai Jiao Tong University http://wnzhang.net http://wnzhang.net/teaching/cs420/index.html ML Task: Function Approximation Problem setting
EECS 349:Machine Learning Bryan Pardo
EECS 349:Machine Learning Bryan Pardo Topic 2: Decision Trees (Includes content provided by: Russel & Norvig, D. Downie, P. Domingos) 1 General Learning Task There is a set of possible examples Each example
Decision Tree Learning and Inductive Inference
Decision Tree Learning and Inductive Inference 1 Widely used method for inductive inference Inductive Inference Hypothesis: Any hypothesis found to approximate the target function well over a sufficiently
Decision Tree Analysis for Classification Problems. Entscheidungsunterstützungssysteme SS 18
Decision Tree Analysis for Classification Problems Entscheidungsunterstützungssysteme SS 18 Supervised segmentation An intuitive way of thinking about extracting patterns from data in a supervised manner
Induction on Decision Trees
Séance «IDT» de l'ue «apprentissage automatique» Bruno Bouzy bruno.bouzy@parisdescartes.fr www.mi.parisdescartes.fr/~bouzy Outline Induction task ID3 Entropy (disorder) minimization Noise Unknown attribute
Learning Classification Trees. Sargur Srihari
Learning Classification Trees Sargur srihari@cedar.buffalo.edu 1 Topics in CART CART as an adaptive basis function model Classification and Regression Tree Basics Growing a Tree 2 A Classification Tree
the tree till a class assignment is reached
Decision Trees Decision Tree for Playing Tennis Prediction is done by sending the example down Prediction is done by sending the example down the tree till a class assignment is reached Definitions Internal
Decision Trees Part 1. Rao Vemuri University of California, Davis
Decision Trees Part 1 Rao Vemuri University of California, Davis Overview What is a Decision Tree Sample Decision Trees How to Construct a Decision Tree Problems with Decision Trees Classification Vs Regression
Machine Learning 2nd Edi7on
Lecture Slides for INTRODUCTION TO Machine Learning 2nd Edi7on CHAPTER 9: Decision Trees ETHEM ALPAYDIN The MIT Press, 2010 Edited and expanded for CS 4641 by Chris Simpkins alpaydin@boun.edu.tr h1p://www.cmpe.boun.edu.tr/~ethem/i2ml2e
Data classification (II)
Lecture 4: Data classification (II) Data Mining - Lecture 4 (2016) 1 Outline Decision trees Choice of the splitting attribute ID3 C4.5 Classification rules Covering algorithms Naïve Bayes Classification
Classification and regression trees
Classification and regression trees Pierre Geurts p.geurts@ulg.ac.be Last update: 23/09/2015 1 Outline Supervised learning Decision tree representation Decision tree learning Extensions Regression trees
Symbolic methods in TC: Decision Trees
Symbolic methods in TC: Decision Trees ML for NLP Lecturer: Kevin Koidl Assist. Lecturer Alfredo Maldonado https://www.cs.tcd.ie/kevin.koidl/cs0/ kevin.koidl@scss.tcd.ie, maldonaa@tcd.ie 01-017 A symbolic
Lecture 3: Decision Trees
Lecture 3: Decision Trees Cognitive Systems II - Machine Learning SS 2005 Part I: Basic Approaches of Concept Learning ID3, Information Gain, Overfitting, Pruning Lecture 3: Decision Trees p. Decision
Decision Support. Dr. Johan Hagelbäck.
Decision Support Dr. Johan Hagelbäck johan.hagelback@lnu.se http://aiguy.org Decision Support One of the earliest AI problems was decision support The first solution to this problem was expert systems
Decision Trees.
. Machine Learning Decision Trees Prof. Dr. Martin Riedmiller AG Maschinelles Lernen und Natürlichsprachliche Systeme Institut für Informatik Technische Fakultät Albert-Ludwigs-Universität Freiburg riedmiller@informatik.uni-freiburg.de
Decision Tree Learning
Topics Decision Tree Learning Sattiraju Prabhakar CS898O: DTL Wichita State University What are decision trees? How do we use them? New Learning Task ID3 Algorithm Weka Demo C4.5 Algorithm Weka Demo Implementation
DECISION TREE LEARNING. [read Chapter 3] [recommended exercises 3.1, 3.4]
![DECISION TREE LEARNING. [read Chapter 3] [recommended exercises 3.1, 3.4]](/thumbs/78/78784288.jpg "DECISION TREE LEARNING. [read Chapter 3] [recommended exercises 3.1, 3.4]")
1 DECISION TREE LEARNING [read Chapter 3] [recommended exercises 3.1, 3.4] Decision tree representation ID3 learning algorithm Entropy, Information gain Overfitting Decision Tree 2 Representation: Tree-structured
Data Mining. CS57300 Purdue University. Bruno Ribeiro. February 8, 2018
Data Mining CS57300 Purdue University Bruno Ribeiro February 8, 2018 Decision trees Why Trees? interpretable/intuitive, popular in medical applications because they mimic the way a doctor thinks model
Machine Learning 3. week
Machine Learning 3. week Entropy Decision Trees ID3 C4.5 Classification and Regression Trees (CART) 1 What is Decision Tree As a short description, decision tree is a data classification procedure which
Decision Tree Learning Mitchell, Chapter 3. CptS 570 Machine Learning School of EECS Washington State University
Decision Tree Learning Mitchell, Chapter 3 CptS 570 Machine Learning School of EECS Washington State University Outline Decision tree representation ID3 learning algorithm Entropy and information gain
Introduction to ML. Two examples of Learners: Naïve Bayesian Classifiers Decision Trees
Introduction to ML Two examples of Learners: Naïve Bayesian Classifiers Decision Trees Why Bayesian learning? Probabilistic learning: Calculate explicit probabilities for hypothesis, among the most practical
Classification and Prediction
Classification Classification and Prediction Classification: predict categorical class labels Build a model for a set of classes/concepts Classify loan applications (approve/decline) Prediction: model
Introduction to Machine Learning CMU-10701
Introduction to Machine Learning CMU-10701 23. Decision Trees Barnabás Póczos Contents Decision Trees: Definition + Motivation Algorithm for Learning Decision Trees Entropy, Mutual Information, Information
Lecture 3: Decision Trees
Lecture 3: Decision Trees Cognitive Systems - Machine Learning Part I: Basic Approaches of Concept Learning ID3, Information Gain, Overfitting, Pruning last change November 26, 2014 Ute Schmid (CogSys,
Classification: Rule Induction Information Retrieval and Data Mining. Prof. Matteo Matteucci
Classification: Rule Induction Information Retrieval and Data Mining Prof. Matteo Matteucci What is Rule Induction? The Weather Dataset 3 Outlook Temp Humidity Windy Play Sunny Hot High False No Sunny
Induction of Decision Trees
Induction of Decision Trees Peter Waiganjo Wagacha This notes are for ICS320 Foundations of Learning and Adaptive Systems Institute of Computer Science University of Nairobi PO Box 30197, 00200 Nairobi.
Dan Roth 461C, 3401 Walnut
CIS 519/419 Applied Machine Learning www.seas.upenn.edu/~cis519 Dan Roth danroth@seas.upenn.edu http://www.cis.upenn.edu/~danroth/ 461C, 3401 Walnut Slides were created by Dan Roth (for CIS519/419 at Penn
Machine Learning Alternatives to Manual Knowledge Acquisition
Machine Learning Alternatives to Manual Knowledge Acquisition Interactive programs which elicit knowledge from the expert during the course of a conversation at the terminal. Programs which learn by scanning
Decision Trees. Danushka Bollegala
Decision Trees Danushka Bollegala Rule-based Classifiers In rule-based learning, the idea is to learn a rule from train data in the form IF X THEN Y (or a combination of nested conditions) that explains
Notes on Machine Learning for and
Notes on Machine Learning for 16.410 and 16.413 (Notes adapted from Tom Mitchell and Andrew Moore.) Learning = improving with experience Improve over task T (e.g, Classification, control tasks) with respect
Decision Trees.
. Machine Learning Decision Trees Prof. Dr. Martin Riedmiller AG Maschinelles Lernen und Natürlichsprachliche Systeme Institut für Informatik Technische Fakultät Albert-Ludwigs-Universität Freiburg riedmiller@informatik.uni-freiburg.de
Machine Learning 2010
Machine Learning 2010 Decision Trees Email: mrichter@ucalgary.ca -- 1 - Part 1 General -- 2 - Representation with Decision Trees (1) Examples are attribute-value vectors Representation of concepts by labeled
Symbolic methods in TC: Decision Trees
Symbolic methods in TC: Decision Trees ML for NLP Lecturer: Kevin Koidl Assist. Lecturer Alfredo Maldonado https://www.cs.tcd.ie/kevin.koidl/cs4062/ kevin.koidl@scss.tcd.ie, maldonaa@tcd.ie 2016-2017 2
Lecture 24: Other (Non-linear) Classifiers: Decision Tree Learning, Boosting, and Support Vector Classification Instructor: Prof. Ganesh Ramakrishnan
Lecture 24: Other (Non-linear) Classifiers: Decision Tree Learning, Boosting, and Support Vector Classification Instructor: Prof Ganesh Ramakrishnan October 20, 2016 1 / 25 Decision Trees: Cascade of step
Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen. Intelligent Data Analysis. Decision Trees
Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen Intelligent Data Analysis Decision Trees Paul Prasse, Niels Landwehr, Tobias Scheffer Decision Trees One of many applications:
CS6375: Machine Learning Gautam Kunapuli. Decision Trees
Gautam Kunapuli Example: Restaurant Recommendation Example: Develop a model to recommend restaurants to users depending on their past dining experiences. Here, the features are cost (x ) and the user s
Tools of AI. Marcin Sydow. Summary. Machine Learning
Machine Learning Outline of this Lecture Motivation for Data Mining and Machine Learning Idea of Machine Learning Decision Table: Cases and Attributes Supervised and Unsupervised Learning Classication
Question of the Day. Machine Learning 2D1431. Decision Tree for PlayTennis. Outline. Lecture 4: Decision Tree Learning
Question of the Day Machine Learning 2D1431 How can you make the following equation true by drawing only one straight line? 5 + 5 + 5 = 550 Lecture 4: Decision Tree Learning Outline Decision Tree for PlayTennis
Decision Trees. Data Science: Jordan Boyd-Graber University of Maryland MARCH 11, Data Science: Jordan Boyd-Graber UMD Decision Trees 1 / 1
Decision Trees Data Science: Jordan Boyd-Graber University of Maryland MARCH 11, 2018 Data Science: Jordan Boyd-Graber UMD Decision Trees 1 / 1 Roadmap Classification: machines labeling data for us Last
Machine Learning Recitation 8 Oct 21, Oznur Tastan
Machine Learning 10601 Recitation 8 Oct 21, 2009 Oznur Tastan Outline Tree representation Brief information theory Learning decision trees Bagging Random forests Decision trees Non linear classifier Easy
Decision Trees. CS57300 Data Mining Fall Instructor: Bruno Ribeiro
Decision Trees CS57300 Data Mining Fall 2016 Instructor: Bruno Ribeiro Goal } Classification without Models Well, partially without a model } Today: Decision Trees 2015 Bruno Ribeiro 2 3 Why Trees? } interpretable/intuitive,
Decision Trees. Gavin Brown
Decision Trees Gavin Brown Every Learning Method has Limitations Linear model? KNN? SVM? Explain your decisions Sometimes we need interpretable results from our techniques. How do you explain the above
Classification Using Decision Trees
Classification Using Decision Trees 1. Introduction Data mining term is mainly used for the specific set of six activities namely Classification, Estimation, Prediction, Affinity grouping or Association
Decision Tree Learning Lecture 2
Machine Learning Coms-4771 Decision Tree Learning Lecture 2 January 28, 2008 Two Types of Supervised Learning Problems (recap) Feature (input) space X, label (output) space Y. Unknown distribution D over
Decision Tree Learning
Decision Tree Learning Berlin Chen Department of Computer Science & Information Engineering National Taiwan Normal University References: 1. Machine Learning, Chapter 3 2. Data Mining: Concepts, Models,
Supervised Learning! Algorithm Implementations! Inferring Rudimentary Rules and Decision Trees!
Supervised Learning! Algorithm Implementations! Inferring Rudimentary Rules and Decision Trees! Summary! Input Knowledge representation! Preparing data for learning! Input: Concept, Instances, Attributes"
Classification and Regression Trees
Classification and Regression Trees Ryan P Adams So far, we have primarily examined linear classifiers and regressors, and considered several different ways to train them When we ve found the linearity
From statistics to data science. BAE 815 (Fall 2017) Dr. Zifei Liu
From statistics to data science BAE 815 (Fall 2017) Dr. Zifei Liu Zifeiliu@ksu.edu Why? How? What? How much? How many? Individual facts (quantities, characters, or symbols) The Data-Information-Knowledge-Wisdom
M chi h n i e n L e L arni n n i g Decision Trees Mac a h c i h n i e n e L e L a e r a ni n ng
1 Decision Trees 2 Instances Describable by Attribute-Value Pairs Target Function Is Discrete Valued Disjunctive Hypothesis May Be Required Possibly Noisy Training Data Examples Equipment or medical diagnosis
Decision Trees (Cont.)
Decision Trees (Cont.) R&N Chapter 18.2,18.3 Side example with discrete (categorical) attributes: Predicting age (3 values: less than 30, 30-45, more than 45 yrs old) from census data. Attributes (split
Introduction. Decision Tree Learning. Outline. Decision Tree 9/7/2017. Decision Tree Definition
Introduction Decision Tree Learning Practical methods for inductive inference Approximating discrete-valued functions Robust to noisy data and capable of learning disjunctive expression ID3 earch a completely
( D) I(2,3) I(4,0) I(3,2) weighted avg. of entropies
Decision Tree Induction using Information Gain Let I(x,y) as the entropy in a dataset with x number of class 1(i.e., play ) and y number of class (i.e., don t play outcomes. The entropy at the root, i.e.,
Machine Learning & Data Mining
Group M L D Machine Learning M & Data Mining Chapter 7 Decision Trees Xin-Shun Xu @ SDU School of Computer Science and Technology, Shandong University Top 10 Algorithm in DM #1: C4.5 #2: K-Means #3: SVM
Learning Decision Trees
Learning Decision Trees CS194-10 Fall 2011 Lecture 8 CS194-10 Fall 2011 Lecture 8 1 Outline Decision tree models Tree construction Tree pruning Continuous input features CS194-10 Fall 2011 Lecture 8 2
Decision Tree Learning
0. Decision Tree Learning Based on Machine Learning, T. Mitchell, McGRAW Hill, 1997, ch. 3 Acknowledgement: The present slides are an adaptation of slides drawn by T. Mitchell PLAN 1. Concept learning:
Decision Trees Entropy, Information Gain, Gain Ratio
Changelog: 14 Oct, 30 Oct Decision Trees Entropy, Information Gain, Gain Ratio Lecture 3: Part 2 Outline Entropy Information gain Gain ratio Marina Santini Acknowledgements Slides borrowed and adapted
UVA CS 4501: Machine Learning
UVA CS 4501: Machine Learning Lecture 21: Decision Tree / Random Forest / Ensemble Dr. Yanjun Qi University of Virginia Department of Computer Science Where are we? è Five major sections of this course
Decision Trees. Nicholas Ruozzi University of Texas at Dallas. Based on the slides of Vibhav Gogate and David Sontag
Decision Trees Nicholas Ruozzi University of Texas at Dallas Based on the slides of Vibhav Gogate and David Sontag Supervised Learning Input: labelled training data i.e., data plus desired output Assumption:
Modern Information Retrieval
Modern Information Retrieval Chapter 8 Text Classification Introduction A Characterization of Text Classification Unsupervised Algorithms Supervised Algorithms Feature Selection or Dimensionality Reduction
Rule Generation using Decision Trees
Rule Generation using Decision Trees Dr. Rajni Jain 1. Introduction A DT is a classification scheme which generates a tree and a set of rules, representing the model of different classes, from a given
The Quadratic Entropy Approach to Implement the Id3 Decision Tree Algorithm
Journal of Computer Science and Information Technology December 2018, Vol. 6, No. 2, pp. 23-29 ISSN: 2334-2366 (Print), 2334-2374 (Online) Copyright The Author(s). All Rights Reserved. Published by American
Mining Classification Knowledge
Mining Classification Knowledge Remarks on NonSymbolic Methods JERZY STEFANOWSKI Institute of Computing Sciences, Poznań University of Technology SE lecture revision 2013 Outline 1. Bayesian classification
Decision Tree. Decision Tree Learning. c4.5. Example
Decision ree Decision ree Learning s of systems that learn decision trees: c4., CLS, IDR, ASSISA, ID, CAR, ID. Suitable problems: Instances are described by attribute-value couples he target function has
CLASSIFICATION NAIVE BAYES. NIKOLA MILIKIĆ UROŠ KRČADINAC
CLASSIFICATION NAIVE BAYES NIKOLA MILIKIĆ nikola.milikic@fon.bg.ac.rs UROŠ KRČADINAC uros@krcadinac.com WHAT IS CLASSIFICATION? A supervised learning task of determining the class of an instance; it is
SUPERVISED LEARNING: INTRODUCTION TO CLASSIFICATION
SUPERVISED LEARNING: INTRODUCTION TO CLASSIFICATION 1 Outline Basic terminology Features Training and validation Model selection Error and loss measures Statistical comparison Evaluation measures 2 Terminology
ML techniques. symbolic techniques different types of representation value attribute representation representation of the first order
MACHINE LEARNING Definition 1: Learning is constructing or modifying representations of what is being experienced [Michalski 1986], p. 10 Definition 2: Learning denotes changes in the system That are adaptive
Administration. Chapter 3: Decision Tree Learning (part 2) Measuring Entropy. Entropy Function
Administration Chapter 3: Decision Tree Learning (part 2) Book on reserve in the math library. Questions? CS 536: Machine Learning Littman (Wu, TA) Measuring Entropy Entropy Function S is a sample of training
Decision trees COMS 4771
Decision trees COMS 4771 1. Prediction functions (again) Learning prediction functions IID model for supervised learning: (X 1, Y 1),..., (X n, Y n), (X, Y ) are iid random pairs (i.e., labeled examples).
Administrative notes. Computational Thinking ct.cs.ubc.ca
Administrative notes Labs this week: project time. Remember, you need to pass the project in order to pass the course! (See course syllabus.) Clicker grades should be on-line now Administrative notes March
CS145: INTRODUCTION TO DATA MINING
CS145: INTRODUCTION TO DATA MINING 4: Vector Data: Decision Tree Instructor: Yizhou Sun yzsun@cs.ucla.edu October 10, 2017 Methods to Learn Vector Data Set Data Sequence Data Text Data Classification Clustering
CS 380: ARTIFICIAL INTELLIGENCE
CS 380: ARTIFICIAL INTELLIGENCE MACHINE LEARNING 11/11/2013 Santiago Ontañón santi@cs.drexel.edu https://www.cs.drexel.edu/~santi/teaching/2013/cs380/intro.html Summary so far: Rational Agents Problem
Inductive Learning. Chapter 18. Why Learn?
Inductive Learning Chapter 18 Material adopted from Yun Peng, Chuck Dyer, Gregory Piatetsky-Shapiro & Gary Parker Why Learn? Understand and improve efficiency of human learning Use to improve methods for
Data Mining Classification: Basic Concepts, Decision Trees, and Model Evaluation
Data Mining Classification: Basic Concepts, Decision Trees, and Model Evaluation Lecture Notes for Chapter 4 Part I Introduction to Data Mining by Tan, Steinbach, Kumar Adapted by Qiang Yang (2010) Tan,Steinbach,
Classification: Decision Trees
Classification: Decision Trees These slides were assembled by Byron Boots, with grateful acknowledgement to Eric Eaton and the many others who made their course materials freely available online. Feel
Tutorial 6. By:Aashmeet Kalra
Tutorial 6 By:Aashmeet Kalra AGENDA Candidate Elimination Algorithm Example Demo of Candidate Elimination Algorithm Decision Trees Example Demo of Decision Trees Concept and Concept Learning A Concept
Chapter 3: Decision Tree Learning
Chapter 3: Decision Tree Learning CS 536: Machine Learning Littman (Wu, TA) Administration Books? New web page: http://www.cs.rutgers.edu/~mlittman/courses/ml03/ schedule lecture notes assignment info.
CLUe Training An Introduction to Machine Learning in R with an example from handwritten digit recognition
CLUe Training An Introduction to Machine Learning in R with an example from handwritten digit recognition Ad Feelders Universiteit Utrecht Department of Information and Computing Sciences Algorithmic Data
Lecture 4 Discriminant Analysis, k-nearest Neighbors
Lecture 4 Discriminant Analysis, k-nearest Neighbors Fredrik Lindsten Division of Systems and Control Department of Information Technology Uppsala University. Email: fredrik.lindsten@it.uu.se fredrik.lindsten@it.uu.se
Decision Trees. CS 341 Lectures 8/9 Dan Sheldon
Decision rees CS 341 Lectures 8/9 Dan Sheldon Review: Linear Methods Y! So far, we ve looked at linear methods! Linear regression! Fit a line/plane/hyperplane X 2 X 1! Logistic regression! Decision boundary
brainlinksystem.com $25+ / hr AI Decision Tree Learning Part I Outline Learning 11/9/2010 Carnegie Mellon
I Decision Tree Learning Part I brainlinksystem.com $25+ / hr Illah Nourbakhsh s version Chapter 8, Russell and Norvig Thanks to all past instructors Carnegie Mellon Outline Learning and philosophy Induction
Decision Trees. CSC411/2515: Machine Learning and Data Mining, Winter 2018 Luke Zettlemoyer, Carlos Guestrin, and Andrew Moore
Decision Trees Claude Monet, The Mulberry Tree Slides from Pedro Domingos, CSC411/2515: Machine Learning and Data Mining, Winter 2018 Luke Zettlemoyer, Carlos Guestrin, and Andrew Moore Michael Guerzhoy
Statistical Learning. Philipp Koehn. 10 November 2015
Statistical Learning Philipp Koehn 10 November 2015 Outline 1 Learning agents Inductive learning Decision tree learning Measuring learning performance Bayesian learning Maximum a posteriori and maximum
Decision-Tree Learning. Chapter 3: Decision Tree Learning. Classification Learning. Decision Tree for PlayTennis
Decision-Tree Learning Chapter 3: Decision Tree Learning CS 536: Machine Learning Littman (Wu, TA) [read Chapter 3] [some of Chapter 2 might help ] [recommended exercises 3.1, 3.2] Decision tree representation
Algorithms for Classification: The Basic Methods
Algorithms for Classification: The Basic Methods Outline Simplicity first: 1R Naïve Bayes 2 Classification Task: Given a set of pre-classified examples, build a model or classifier to classify new cases.
Imagine we ve got a set of data containing several types, or classes. E.g. information about customers, and class=whether or not they buy anything.
Decision Trees Defining the Task Imagine we ve got a set of data containing several types, or classes. E.g. information about customers, and class=whether or not they buy anything. Can we predict, i.e
Einführung in Web- und Data-Science
Einführung in Web- und Data-Science Prof. Dr. Ralf Möller Universität zu Lübeck Institut für Informationssysteme Tanya Braun (Übungen) Inductive Learning Chapter 18/19 Chapters 3 and 4 Material adopted
Unsupervised Learning. k-means Algorithm
Unsupervised Learning Supervised Learning: Learn to predict y from x from examples of (x, y). Performance is measured by error rate. Unsupervised Learning: Learn a representation from exs. of x. Learn
CC283 Intelligent Problem Solving 28/10/2013
Machine Learning What is the research agenda? How to measure success? How to learn? Machine Learning Overview Unsupervised Learning Supervised Learning Training Testing Unseen data Data Observed x 1 x
Numerical Learning Algorithms
Numerical Learning Algorithms Example SVM for Separable Examples.......................... Example SVM for Nonseparable Examples....................... 4 Example Gaussian Kernel SVM...............................
Statistical Machine Learning from Data
Samy Bengio Statistical Machine Learning from Data 1 Statistical Machine Learning from Data Ensembles Samy Bengio IDIAP Research Institute, Martigny, Switzerland, and Ecole Polytechnique Fédérale de Lausanne