Basic Probability and Information Theory: quick revision

Transcription

1 Basic Probability and Information Theory: quick revision ML for NLP Lecturer: S Luz February 17, 2015 In these notes we review the basics of probability theory and introduce the information theoretic notions which are essential to many aspects of machine learning in general, and to the induction of text classifiers in particular. We will attempt to illustrate the abstract concepts introduced in these notes with examples from a text categorization task (e.g. spam filtering). The reader should note, however, that the techniques and results reported here apply to a much wider domain of applications. The connection with TC will become clearer next week when we will see feature selection and classifier induction. Concise but useful introductions to probability theory can be found in (Russell and Norvig, 1995, chapter 14) and (Manning and Schütze, 1999, chapter 2). The latter also includes the basics of information theory, viewed mainly from a natural language processing perspective. A very good general introduction to probability theory is (Bertsekas and Tsitsiklis, 2002). Why review Probability and Information Theory? Probability theory gives us a tool to model uncertainty Probabilistic approaches (e.g. naive Bayes) are used in TC. Information Theory plays an important role in various areas of machine learning, in particular: Feature selection uses the information theoretic notions of information gain and mutual information 1

2 Learning algorithms, such as the decision tree induction, use the information theoretic concept of entropy to decide how to partition the document space. Information theory originated from Claude Shannon s research on the capacity of noisy information channels. Information theory is concerned with maximising the information one can transmit over an imperfect communication channel. The central concept of Information Theory is that of Entropy. Entropy (which we will define formally below) measures the amount of uncertainty in a probability distribuition. Reportedly, the term entropy was suggested to Shannon by John von Newman : You should call it entropy for two reasons: first, the function is already in use in thermodynamics under the same name; second, and more importantly, most people don t know what entropy really is, and if you use the word entropy in an argument you will win every time (Hamming, 1991). Probability theory: notation Notation Set Jargon Probability jargon Ω collection of objects sample space ω element of Ω elementary event D subset of Ω event that some outcome in D occurs D complement of D event that no outcome in D occurs D E intersection both D and E D E union D or E, or both D\E difference D but not E D E inclusion if D then E empty set impossible event Ω whole space certain event A notational variant to the above which stresses the connection with logic would treat set intersection as conjunction, set union as disjunction, etc. This variant is summarised below: Logic Set theory P (A B) P (A B) P (A B) P (A B) P (false) P ( ) P (true) P (Ω) 2

3 Sample spaces The set of all possible outcomes of an experiment is called sample space in Text Categorisation, for instance, one could regard the set of documents being processed as the sample space: Ω = D = {d 1,..., d Ω } However, one could alternatively take sets of documents rather than sets (or multisets, lists etc) of words to be elementary events. In this case, Ω = 2 D would be the sample space. An experiment could be performed to determine, for instance, which documents belong to category c (say, that should be classified as spam). The outcome of that experiment would be a subset of Ω. Different ways of characterising sample spaces in TC will be presented in the lecture on Naive Bayes Text Categorisation. Here is an even more mundane example: the combinations of heads (H) and tails (T) resulting from tossing a coin three times can be represented by the following sample space: Ω = {HHH, HHT, HT H, HT T, T HH, T T H, T HT, T T T } Discrete Uniform Probability Law Now that we have characterised the sample space, we would like to be able to quantify the likelihood of events. Discrete uniform probability law: If the sample space consists of n possible outcomes which are equally likely, then the probability of any event D is given by P (D) = No. of elements of D n One can think of the probability of occurrence of an event D as the proportion of times event D occurs in a large number of trials: P (D) = No. of occurrences of D No. of trials (1) 3

4 The view of probabilities implicit in the view adopted above has been termed frequentist. It relies on the empirical observation that the ratio between observed occurrences of an event and the number of trials appears to converge to a limit as the number of trials increases. The frequentist approach is inadequate in many ways, but a thorought discussion of its merits and limitations is beyond the scope of this revision. For a very readable discussion of the philosophy of probability theory, see (Hamming, 1991). We may illustrate this approach by calculating the probability that a document of corpus Ω is filed under category c as follows: where D = {ω Ω : f(ω, c) = T } P (c) = D Ω Visualisating sample spaces Sample spaces can be depicted in different ways. For events described by two rolls of a die, the sample space could be depicted as a grid: E2= {at least one roll is 5} In experiments of a sequential nature such as this, a tree representation is also informative.... 2nd roll... 1st roll E1= {same result in both rolls} P (E 1 ) = 1 6, P (E 2) = st roll nd roll (adapted from (Bertsekas and Tsitsiklis, 2002)). Sample space, in somewhat more formal terms Defining σ-field: 4

5 Example: A collection F of subsets of Ω is called a σ-field if it satisfies the following conditions: 1. F 2. if D 1,..., D n F then n i=1 D i F 3. if D F then D F the smallest σ-field associated with Ω... is the collection F = {,Ω}. Probability spaces We continue to add structure to our original set of events by defining P as a probability measure: A Probability Measure P on < Ω, F > is a function P : F [0, 1] satisfying: 1. P (Ω) = 1 2. if D 1, D 2,... is a collection of disjoint members of F, in that D i D j = for all i j, then P ( D i ) = i=1 P (D i ) i=1 The triple < Ω, F, P > is called a Probability Space In the definition of probability measure presented here, F is obviously to be understood as a σ-field. A probability measure is a special case of what is called in probability theory simply a measure. A measure is a function µ : F [0, ), satisfying (2) as above and µ( ) = 0. Some weight assignment functions, such as the ones often used in the decision trees are measures, though they are not probability measures. 5

6 Properties of probability spaces The following hold: P ( D) = 1 P (D) (2) If E D then P (E) = P (D) + P (E\D) P (D) (3) P (A B C) = P (A) + P (Ā B) + P (Ā B C) (4) Inclusion-exclusion principle: or, more generally: Proofs: P (D E) = P (D) + P (E) P (D E) (5) n P ( D i ) = P (D i ) P (D i D j ) i=1 i=1 i<j + P (D i D j D k )... i<j<k +( 1) n+1 P (D 1... D n ) (2) D D = Ω and D D =, so P (D D) = P (D) + P ( D) = 1 (3) If E D, then E = D (E\D), which is a union of disjoint sets. Therefore P (E) = P (D) + P (E\D) (5) The rationale for the Inclusion-exclusion principle is easy to visualise by drawing a Venn diagram of (possibly intersecting) sets D and E. Simply adding the probabilities of D and E is as if we are counting the probability of the intersection twice, so the result needs to be readjusted by subtracting the intersection: P (D E) = P ((D\E) (D E) (E\D)) (set theory) = P (D\E) + P (D E) + P (E\D) (disjoint sets) = P (D\E) + P (D E) + P (E\D) + P (D E) P (D E) (algebra) = P ((D\E) (D E)) + P ((E\D) (D E)) P (D E) (disjoint sets) = P (D) + P (E) P (D E) (set theory) 6

7 Conditional probability If P (E) > 0, then the conditional probability that D occurs given E is defined to be: P (D E) P (D E) = P (E) The diagram above illustrates conditional probabilities in terms of set size: If you see probability measures as frequencies/proportion of occurrence, then the conditional is given by D E E = D E Ω E Ω Properties of conditional probabilities = P (D E) P (E). 1. For any events D and E s.t. 0 < P (E) < 1, P (D) = P (D E)P (E) + P (D Ē)P (Ē) 2. More generally, if E 1,..., E n are partitions of Ω s.t. P (E i ) > 0, then n P (D) = P (D E i )P (E i ) 3. Chain rule: Proof: i=1 P (D 1... D n ) = P (D 1 )P (D 2 D 1 )P (D 3 D 2 D 1 ) D = (E D) (Ē D), which is a union of disjoint sets. Thus P (D) = P (E D) + P (Ē D) = P (D E)P (E) + P (D Ē)P (Ē) 7

8 Bayes rule Sometimes, as in the case of naive Bayes TC, it is easier to estimate the conditional probability of E given D than the other way around. In such cases, Bayes rule can be used to simplify computation: P (E D) = = P (E D) P (D) P (D E)P (E) P (D) (6) Proof: It follows trivially from the definition of conditional probability (slide 10) and the chain rule (slide 11). Independence In general, the occurrence of an event E changes the probability that another event D occurs. When this happens, the initial (prior) probability P (D) gets updated to P (D E). If the probability remains unchanged, i.e. P (D) = P (D E), then we call D and E independent: Events D 1 and D n are called independent if ( ) P D i = P (D i ) S {1, 2,..., n} i S i S E.g: Two fair coin tosses are independent. But note that, when we have more than two events, pairwise indepence does not imply indepence: from P (C A) = P (C) and P (C B) = P (C) you cannot conclude P (A B C) = P (A)P (B)P (C) Neither is the latter a sufficient condition for the indepence of A, B and C Examples (verify that they are actually the case, as an exercise): 1. Pairwise indepence does not imply indepence: A = { coin comes up heads on the first toss } B = { coin comes up heads on the second toss } C = { the two tosses have different results } 8

9 2. P (A B C) = P (A)P (B)P (C) is not sufficient for independence: Consider two throws of a fair die and the following events: A = { first roll is 1, 2 or 3 } B = { first roll is 3, 4, or 5 } C = { the sum of the two rolls is 9 } 3. Similarly for a set of random variables S = {X 1,..., X n }, P (X i Xj S\{X i } X j ) = P (X i ) does not imply independence for S: Again, consider two throws of a fair die and the following events: A = { first roll is 1, 3 or 4 } B = { first roll is 1, 2, or 4 } C = { the sum of the two rolls is 4 } (show that P (A B C) = P (A) and P (B A C) = P (B) and P (C B A) = P (C) but P (A B) P (A) etc) Conditional Independence Absolute independence (as described above) is a very strong requirement, which is seldom met. In practice, one often uses conditional independence: P (A B C) = P (A C)P (B C) (7) or, equivalently: P (A B C) = P (A C) (8) E.g.: Let A and B be two biased coins in that probab. of heads for A is.99 and for B.01. Choose a coin randomly (with a.5 probab. of choosing each) and toss it twice. The probabily of heads in the 2nd toss is not independent of the prob. of heads in the 1st, but they are independent given the choice of coin. Some exercises 1. The dangers of overtesting: Domingos (2012) review of ML used the following example to caution readers against overtesting:... a mutual fund that beats the market ten years in a row looks very impressive, until you realize that, if there are 1000 funds and each has a 50% chance of beating the market on any given year, its quite likely that one will succeed all ten times just by luck. 9

10 Question: What is the actual probability that one mutual fund will succeed all 10 times by luck? 2. Monty Hall (from Wikipedia): Suppose you re on a game show, and you re given the choice of three doors: Behind one door is a car; behind the others, goats. You pick a door, say No. 1, and the host, who knows what s behind the doors, opens another door, say No. 3, which has a goat. He then says to you, Do you want to pick door No. 2? Is it to your advantage to switch your choice? Question: The best strategy (as Wikipedia will tell you) is to switch. Give an explanation of why that is the case based on conditional probabilities and Bayes s rule. Random variables A random variable is a function X : Ω R with the property that {ω Ω : X(ω) x} F, for each x R Random variables offer a convenient way of abstracting over event spaces. E.g.: The notation P (X = x) is used to indicate the probability that a random variable X takes value x Another example: in categories over a, category can be seen as a random variable which returns the number of documents classified under a given category. Example (ctd.): Assume that 5 documents, out of a 20-document corpus Ω have been classified as spam. We are now regarding subsets of Ω (possibly the entire power set of Ω) defined by, for instance, the categories assigned to their elements as a σ-field, and the resulting triple < Ω, F, P > as a probability space. Events in such probability space will be things that denote 10

11 elements of F, such as the event that documents have been filed under category C. Lets also assume that category spam denotes a set {d 1,..., d 5 } F. The probability associated with the event that documents have been filed under category C is summarised in the random variable and is given by P (C). The probability associated with the event that documents have been filed under category spam is given by a specific value of the random variable (recall that what we are calling variable here is actually a function) is P (C = spam) = Discrete Random Variables A discrete random variable is a random variable whose range is finite (or countably infinite) A discrete random variables are associated with a probability mass function (PMF) A PMF maps each numerical value that a random variable can take to a probability A function of a discrete random variable defines another discrete random variable The PMF of this new random variable can be obtained from the PMF of the original one. Thus a random variable can be conditioned on another random variable (or on an event), and the notions of independence and conditional independence seen above also apply. Probability mass functions The PMF of a random variable X is the function p : R [0, 1], given by p(x) = P (X = x) For a discrete random variable: i N p(x i ) = i P (A xi ) = P (Ω) = 1 where A xi = {ω Ω : X(ω) = x i } So, to calculate the PMF of X we add the probabilities of all events X = x for each possible value x to get p(x). 11

12 E.g.: If X is the number of heads obtained in two tosses of a fair coin, its PMF is:.25 if x = 0 or x = 2 p(x) =.5 if x = 1 0 otherwise So the probability of at least one head is P (X > 0) =.75 Continuous Random Variables A continuous random variable is a random variable whose range is continous (e.g. velocity, time intervals etc) Variable X is called continous if there is a function f X of X s.t., for every subset B of R: P (X B) = f X (x)dx E.g.: the probability that X falls within interval [a, b] is P (a X b) = b a f(x)dx f X is called the probability density function (PDF) of X provided that it is non-negative and has the normalisation property: B f X (x)dx = P ( X ) = 1 Note that for a single value v, P (X = v) = v v f X(x)dx = 0, so the probability that X falls within interval [a, b] is the same as the probability that X falls within [a, b), (a, b] or (a, b) (i.e. it makes no difference wheter the endpoints are included or not. The probability that X falls within interval [a, b] can be interpreted as the area under the PDF curve for the interval. Cumulative Distribution Functions Cumulative Distribution Functions (CDF) subsume PDFs and PMFs under a single concept. The CDF F X of X gives the probability P (X x) so that for every x: p(k) if X is discrete k x F X (x) = P (X x) = x f X (t)dt if X is continuous 12

13 Since X x is always an event (having therefore a well defined probaility), every random variable X associated with a given probability model has a CDF. Moments, expectation, mean, variance The expected value of a discrete random variable X with PMF p is given by E[X] = x p(x)x For a continuous variable with PDF f we have E[X] = xf(x)dx This is AKA the expectation, mean or the first moment of X. In general, we define the n th moment as E[X n ] The variance of a random variable is defined as the expectation of the random variable (X E[X]) 2 : var(x) = E[(X E[X]) 2 x p(x)(x E[X])2 if X is discrete ] = (x E[X])2 f(x)dx if X is continuous Some Discrete Random Variables Bernoulli (parameter p): success (or failure) in a single trial: { p if k = 1 p(k) = E[X] = p, var(x) = p(1 p) 1 p if k = 0 Binomial (parameters p and n): successes in n independent Bernoulli trials: ( ) n p(k) = p k (1 p) n k, k = 0, 1,..., n k Geometric (parameter p): number of trials until first success: E[X] = np, var(x) = np(1 p) p(k) = (1 1) k 1 p, k = 0, 1,... E[X] = 1 p, var(x) = 1 p p 2 Poisson (parameter λ): approximation of binomial PMF when n is large, p is small and λ = np: p(k) = e λ k 1 λk k! k = 0, 1,... E[X] = λ, var(x) = λ 13

14 Some Continuous Random Variables Uniform (over interval [a, b]): f(x) = { 1 b a, if a x b 0, otherwise E[X] = a + b (b 2)2, var(x) = 2 12 Exponential (parameter λ): e.g. model time until some event occurs: { λe λx, if x 0 f(x) = 0, otherwise Geometric CDF Expoential CDF E[X] = 1 λ, var(x) = 1 λ 2 0 x Normal or Gaussian (parameters µ and σ 2 > 0): f(x) = 1 2πσ e (x µ)2 /2σ 2 E[X] = µ, var(x) = σ 2 Entropy Entropy, AKA self-information, measures the average amount of uncertainty in a probability mass distribution. In other words, entropy is a measure of how much we learn when we observe an event occurring in accordance with that distribution. The entropy of a random variable measures the amount of information in that variable (we will always be using log base 2 unless stated 14

15 otherwise): N.B.: We define 0 log(0) = 0 H(X) = H(p) = x X p(x) log p(x) = x X p(x) log 1 p(x) Example: suppose we have a set of documents D = {d 1,..., d n }, each classified according to whether it belongs or not to a certain category c, say, spam. First, suppose you know that all documents in D are filed under spam (we represent that as P (spam) = 1). How much information would we gain if someone told us that a certain document d i drawn randomly from corpus D has been filed under spam? Answer: zero, since we already knew this from the start! Now suppose that you know 80% of D (you incoming folder) is spam, and you randomly pick an message from D and find out that it is labelled spam. How much have you learned? Certainly more than before, although less than you would have learned if the proportion between spam and legitimate were In the former case there was less uncertainty involved than in the latter. Information gain We may also quantify the reduction of uncertainty of a random variable due to knowing about another. This is known as Expected Mutual Information: I(X; Y ) = IG(X, Y ) = x,y p(x, y) p(x, y) log p(x)p(y) (9) Entropies of different probability functions may also be compared by calculating the so called Information Gain. In decision tree learning, for instance: n G(D, F ) = H(t) p i H(t i ) (10) where t is the distribution of the mother, t i the distribution of daughter node i, and p i the proportion of texts passed to node i if term F is used to split corpus D. i=1 15

16 Information theory tries to quantify such uncertainties by assuming that the amount of information learned from an event is inversely proportional to the probability of its occurrence. So, in the case where there s a chance that d i will be spam, the amount learned from d i would be 1 i(p (C = spam)) = log P (d i = spam) = log = 1 i(.), as defined above, measures the uncertainty for a single value of random variable C. How would we measure uncertainty for all possible values of C (in this case, {spam, spam})? The answer is: we calculate the entropy of its probability mass function: H(C) = (p(spam) log p(spam) + p(spam)) log p(spam) = (0.5 log log 0.5) = 1 A more interesting corpus, where the probability of a document being labelled as spam is, say 0.25, would have entropy of H(C) = (0.25 log log 0.75) = N.B.: There s some confusion surrounding the related notions of Expected Mutual Information and Information Gain. The definition in (9) corresponds to what some call Information Gain (Sebastiani, 2002). For the purposes of choosing where to split the instances in decision trees, the definition of Information Gain used is the one in (10), as defined in (Manning and Schütze, 1999, ch. 2). We will reserve the term Expected Mutual Information I(X; Y ) for what Sebastiani (2002) call Information Gain, though we will sometimes write it IG(X, Y ). We will see details of information gain scores are used in decision tree induction when we review the topic next week. References Bertsekas, D. and Tsitsiklis, J. (2002). Introduction to Probability. Athena Scientific. Domingos, P. (2012). A few useful things to know about machine learning. Communications of the ACM, 55(10):

17 Hamming, R. W. (1991). The Art of Probability for Scientists and Engineers. Addison-Wesley. Manning, C. D. and Schütze, H. (1999). Foundations of Statistical Natural Language Processing. The MIT Press, Cambridge, Massachusetts. Russell, S. J. and Norvig, P. (1995). Artificial Intelligence. A Modern Approach. Prentice-Hall, Englewood Cliffs. Sebastiani, F. (2002). Machine learning in automated text categorization. ACM Computing Surveys, 34(1):