Information Theory, Statistics, and Decision Trees

Size: px

Start display at page:

Download "Information Theory, Statistics, and Decision Trees"

Sophie Mason
6 years ago
Views:

1 Information Theory, Statistics, and Decision Trees Léon Bottou COS 424 4/6/2010

2 Summary 1. Basic information theory. 2. Decision trees. 3. Information theory and statistics. Léon Bottou 2/31 COS 424 4/6/2010

3 I. Basic Information theory Léon Bottou 3/31 COS 424 4/6/2010

4 Why do we care? Information theory Invented by Claude Shannon in 1948 A Mathematical Theory of Communication. Bell System Technical Journal, October The quantity of information measured in bits. The capacity of a transmission channel. Data coding and data compression. Information gain A derived concept. Quantify how much information we acquire about a phenomenon. A justification for the Kullback-Leibler divergence. Léon Bottou 4/31 COS 424 4/6/2010

5 The coding paradigm Intuition The quantity of information of a message is the length of the smallest code that can represent the message. Paradigm Assume there are n possible messages i = 1... n. We want a signal that indicates the occurrence of one of them. We can transmit an alphabet of r symbols. For instance a wire could carry r = 2 electrical levels. The code for message i is a sequence of l i symbols. Properties Codes should be uniquely decodable. Average code length for a message: n x=1 p i l i. Léon Bottou 5/31 COS 424 4/6/2010

6 Prefix codes Messages 1 and 2 have codes one symbol long (l i = 1). Messages 3 and 4 have codes two symbols long (l i = 2). Messages 5 and 6,have codes three symbols long (l i = 2). There is an unused three symbol code. That s inefficient. Properties Prefix codes are uniquely decodable. There are trickier kinds of uniquely decodable codes, e.g. a 0, b 01, c 011 versus a 0, b 10, c 110. Léon Bottou 6/31 COS 424 4/6/2010

7 Kraft inequality Uniquely decodable codes satisfy n ( ) 1 li 1 r x=1 All uniquely decodable codes satisfy this inequality. If integer code lengths l i satisfy this inequality, there exists a prefix code with such code lengths. Consequences If some messages have short codes, others must have long codes. To minimize the average code length: - give short codes to high probability messages. - give long codes to low probability messages. Equiprobable messages should have similar code lengths. Léon Bottou 7/31 COS 424 4/6/2010

8 Kraft inequality for prefix codes Prefix codes satisfy Kraft inequality r l l i r l i i ( ) 1 li 1 r All uniquely decodable codes satisfy Kraft inequality Proof must deal with infinite sequences of messages. Given integer code lengths l i : Build a balanced r-ary tree of depth l = max i l i. For each message, prune one subtree at depth l i. Kraft inequality ensures that there will be enough branches left to define a code for each message. Léon Bottou 8/31 COS 424 4/6/2010

9 Redundant codes Assume i r l i < 1 There are leftover branches in the tree. There are codes that are not used, or There are multiple codes for each message. For best compression, i r l i = 1 This is not always possible with integer code lengths l i. But we can use this to compute a lower bound. Léon Bottou 9/31 COS 424 4/6/2010

10 Lower bound for the average code length Choose code lengths l i such that min p i l i subject to l 1...l n i i r l i = 1, l i > 0 Define s i = r l i, that is, l i = log r (s i ). Maximize C = p i log r (s i ) subject to i s i = 1 We get C s = p i i s i log(r) = Constant, that is s i p i. Replacing in the constraint gives s i = p i. Therefore l i = log r (p i ) and p i l i = i i p i log r (p i ) Fractional code lengths What does it mean to code a message on 0.5 symbols? Léon Bottou 10/31 COS 424 4/6/2010

11 Arithmetic coding An infinite sequence of messages i 1, i 2,... can be viewed as a number x = 0.i 1 i 2 i 3... in base n. An infinite sequence of symbols c 1, c 2,... can be viewed as a number y = 0.c 1 c 2 c 3... in base r. Léon Bottou 11/31 COS 424 4/6/2010

12 Arithmetic coding To encode a sequence of L messages i 1,..., i L. L The code y must belong to an interval of size p ik. It is sufficient to specify l(i 1 i 2... i L ) = L log r (p ik ) digits of y. Léon Bottou 12/31 COS 424 4/6/2010 k=1 k=1

13 Arithmetic coding To encode a sequence of L messages i 1,..., i L. It is sufficient to specify l(i 1 i 2... i L ) = The average code length per message is 1 L p L i1... p il log r (p ik ) i 1 i 2...i L k=1 L L log r (p ik ) p i1... p il L i 1 i 2...i L = 1 L L k=1 i 1...i L \i k k=1 ( h k p ih ) r L log r (p ik ) digits of y. k=1 i k =1 p ik log p ik = i p i log p i Arithmetic coding reaches the lower bound when L. Léon Bottou 13/31 COS 424 4/6/2010

14 Quantity of information Optimal code length: l i = log r (p i ). Optimal expected code length: p i l i = p i log r (p i ). Receiving a message x with probability p x : The acquired information is h(x) = log 2 (p x ) bits. An informative message is a surprising message! Expecting a message X with distribution p 1... p n : The expected information is H(X) = x X p x log 2 (p x ) bits. This is also called entropy. These are two distinct definitions! Note how we switched to logarithms in base two. This is a multiplicative factor: log 2 (p) = log r (p) log 2 (r). Choosing base 2 defines a unit of information: the bit. Léon Bottou 14/31 COS 424 4/6/2010

15 Mutual information Expected information: H(X) = i P (X = i) log P (X = i) Joint information: H(X, Y ) = i,j P(X = i, Y = j) log P (X = i, Y = j) Mutual information: I(X, Y ) = H(X) + H(Y ) H(X, Y ) Léon Bottou 15/31 COS 424 4/6/2010

16 II. Decision trees Léon Bottou 16/31 COS 424 4/6/2010

17 Car mileage Predict which cars have better mileage than 19mpg. mpg cyl disp hp weight accel year name buick skylark plymouth satellite ford galaxie chevrolet impala amc ambassador dpl plymouth cuda volvo 145e volkswagen peugeot renault ford pinto datsun chrysler new yorker... Léon Bottou 17/31 COS 424 4/6/2010

18 Questions Many questions can distinguish cars How many cylinders? (3,4,5,8) Displacement greater than 200 cu in? (yes, no) Displacement greater than x cu in? (yes, no) Weight greater than x lbs? (yes, no) Model name longer than x characters (yes, no) etc... Which question brings the most information about the task? Build contingency table. Compare mutual informations I(Question, M pg > 19). Possible answers ansa ansb ansc ansd mpg> mpg Léon Bottou 18/31 COS 424 4/6/2010

19 Mutual information Consider a contingency table, x ij. 1 j p refers to the question answers X. 1 i n refers to the target values Y. ansa ansb ansc ansd mpg> mpg Let x i = p j=1 x ij, x j = n i=1 x ij, and x = n i=1 p j=1 x ij. Mutual information: I(X, Y ) = H(X, Y ) + H(X) + H(Y ) = ij x ij x log x ij x j x j x log x j x i x i x log x i x Léon Bottou 19/31 COS 424 4/6/2010

20 Decision stump The question generates a partition of the examples. Now we can repeat the process for each node: build the contingency tables. pick the most informative question. Léon Bottou 20/31 COS 424 4/6/2010

21 Decision trees Until all leafs contain a single car. Léon Bottou 21/31 COS 424 4/6/2010

22 Decision trees Then label each leaf with class MP G > 19 or MP G 19. We can now say if a car does more than 19mpg by asking a few questions. But that is learning by heart! Léon Bottou 22/31 COS 424 4/6/2010

23 Pruning the decision tree We can label each node with its dominant class MP G > 19 or MP G 19. The usual picture. Should we use a validation set? Which stopping criterion? the node depth? the node population? Léon Bottou 23/31 COS 424 4/6/2010

24 The χ 2 independence test We met this test when studying correspondence analysis (lecture 10). x i = p j=1 x ij x j = n i=1 x ij x = n i=1 p j=1 x ij E ij = x i x j x If the rows and columns variables were independent X 2 = (x ij E ij ) 2 would asymptotically follow a χ 2 distribution E ij ij with (n 1)(p 1) degrees of freedom. Léon Bottou 24/31 COS 424 4/6/2010

25 Pruning a decision tree with the χ 2 test We want to prune nodes when the contingency table suggests that there is no dependence between the question and the target class. Compute X 2 = ij (x ij E ij ) 2 E ij for each node. Prune if 1 F χ 2(X) > p. Parameter p could be picked by cross-validation. But choosing p = 0.05 often works well enough. Léon Bottou 25/31 COS 424 4/6/2010

26 Conclusion Good points Decision trees run quickly. Decision trees can handle all kinds of input variables. Decision trees can be interpreted relatively easily. Decision trees can handle lots of irrelevant features. Bad points Decision trees are moderately accurate. Small changes in the training set can lead to very different trees. (were we speaking about interpretability... ) Notes Other names for decision trees: ID3, C4.5, CART. Regression tree when the target is continuous. Léon Bottou 26/31 COS 424 4/6/2010

27 III. Information theory and statistics Léon Bottou 27/31 COS 424 4/6/2010

28 Revisiting decision trees : likelihoods The tree as a model of P (Y X) Estimate P (Y X) by the target frequencies in the leaf for X. We can compute the likelihood of the data in this model. Likelihood gain when splitting a node Let x ij be the contingency table for a node and a question. Splitting the node with a question increases the likelihood: log L after log L before = ij x ij log x ij x j i x i log x i x = ij x ij log x ij x x x j i x i log x i x = ij x ij log x ij x j x j log x j x i x i log x i x Compare with slide 19. Léon Bottou 28/31 COS 424 4/6/2010

29 Revisiting decision trees : log loss The tree as a discriminant function p Define f(x) = log X where p 1 p X is the frequency of positive examples X in the leaf corresponding to X. ( ( log 1 + e yf(x)) log 1 1 p ) X p = log(p = X X ) if y = 1 ( log 1 p ) X 1 p = log(1 p X X ) if y = 1 Log loss reduction when splitting a node Let x ij be the contingency table for a node and a question. R before R after = x i log x i + x x ij log x ij x i j i j = x ij log x ij x x j log x j x x i log x i x ij j i Compare with slides 19 and 28. Note: regression trees use the mean squared loss. Léon Bottou 29/31 COS 424 4/6/2010

30 Kullback Leibler divergence Definition KL divergence between a true distribution P (X) and an estimated distribution P θ (X). D(P P θ ) = log P (x) P θ (x) dp (x) = = P (x) log P θ (x) } x {{} H approx H opt : Optimal coding length for X. x P (x) log P (x) P θ (x) P (x) log P (x) } x {{} H opt H approx : Expected code length for X when the code is designed for distribution P θ instead of the true distribution P. The KL divergence measures the excess coding bits when the code is optimized for the estimated distribution instead of the true distribution. Léon Bottou 30/31 COS 424 4/6/2010

31 Maximum Likelihood Minimize KL divergence min D(P P θ ) = log P (x) θ P θ (x) dp (x) max θ log P θ (x) dp (x) Maximize Log Likelihood max θ 1 n n log P θ (x i ) i=1 The log likelihood estimates Constant D(P P θ ) using the training set. Maximizing the likelihood minimizes an estimate of the excess coding bits obtained by coding the training set. One hopes to achieve a good coding performance on future data. The Vapnik-Chervonenkis theory gives confidence intervals for the deviation ( ) ( ) 1 n log P θ (x) dp (x) log P θ (x i ) n i=1 Léon Bottou 31/31 COS 424 4/6/2010

Expectation-Maximization

Expectation-Maximization Léon Bottou NEC Labs America COS 424 3/9/2010 Agenda Goals Representation Capacity Control Operational Considerations Computational Considerations Classification, clustering, regression,