Randomized Weighted Majority Algorithm And Classification Problem

Transcription

1 CSL 758: Advanced Algorithms Scribe: Sandeep Goyal, Saurabh Agrawal Lecturer: Naveen Garg, Kavitha Telikepalli Date: March 31, 2008 Randomized Weighted Majority Algorithm And Classification Problem 1 Randomized Weighted Majority Algorithm 1.1 Introduction Given: We have 2 horses A and B to bet on and N experts E 1, E 2, E 3,...E N to take advice from. W 1, W 2,..., W N are the corresponding weights of the experts. Aim: To bound the number of mistakes I make as compared to the best expert. The algorithm we discussed in the last lecture, the Weighted Majority algorithm, gives the following bound. m < M log 1 log 1 2 log N log 1 2 where, m is Total number of mistakes made. M is Number of mistakes made by the Best Expert is the facor by which we modify the weights of the experts who predicted wrongly. Special Case 1 : = 1 2 Special Case 2 : very small m M log 2 log log N log 4 3 m 2 M + 2 log N In this lecture we will look at Randomized Weighted Majority Algorithm that will improve the bound on m. 1.2 Algorithm Let us define : W A : Total weight of the experts predicting that horse A will win. W B : Total weight of the experts predicting that horse B will win. : Total weight of all experts on day t Scribe Notes, CSL 758, Advanced Algorithms - 1

2 W i : Total weight of all experts initially, which is N. W f : Total weight of all experts finally F t : Total weight of all experts who predict wrongly on day t : Facor by which we modify F t. Now, we bet on horse A with probability Each time we reduce F t by a factor of, so +1 can be computed as: W A W A +W B +1 = F t = (1 F t W f = W i (1 F t = N (1 F t log W f = log N + log(1 F t (1 Using approximation: log(1 + x = x x 2 / log(1 + x x, ifx << 1 (2 From 1 and 2, we have, log W f log N ( F t ( F t log N log W f (3 Now, if the best expert makes M mistakes then, W f (1 M log W f M log(1 (4 Scribe Notes, CSL 758, Advanced Algorithms - 2

3 ExpectedNumberofMistakes = P robabiltyofmistake 1 = ( F t log N log W f log N log(1 M log N + M (5 This new bound is tighter than the earlier bounds. 2 Classification Problem 2.1 Some Definitions A labeled example is an example together with a labeling e.g. 0 or 1 A concept is a boolean function over an instance space. For instance, the concept x 1 x 2 over {0, 1} n is the boolean function that outputs 1 on any input in which the first two bits are set to 1. Sometimes we will think of a concept c as the set of its positive examples, i.e. x c means that c(x = 1. A concept class is a set of concepts, typically with an associated representation. For instance, the class of monotone conjunctions consists of all concepts that can be expressed as a conjunction of variables 2.2 Consistancy Model We say that algorithm A learns class C in the consistency model if given any set of labeled examples S, the algorithm produces a concept c C consistent with S if one exists, and outputs there is no consistent concept, otherwise. 2.3 Algorithm f 1 f 2 f 3... f n E E E The class of conjunctions is the class of ANDs of literals, where a literal is a variable. E.g. a typical concept might be x 1 x 4 x 6. This class is learnable by the following algorithm: Scribe Notes, CSL 758, Advanced Algorithms - 3

4 Produce the conjunction of all literals that are satisfied by every positive example. (I.e. throw out any literal falsified by some positive example. By definition, this is consistent with all the positive examples. In fact, this is the most specific conjunction consistent with the positive examples because we only throw out literals when absolutely necessary; therefore, if any conjunction is consistent with all the examples, this one is. If the above conjunction is also consistent with the negative examples, produce it as output. Otherwise halt with failure. 2.4 Proof of correctness of Algorithm Case 1: Output conjunction is consistent with the negative examples This conjuction satisfies both the positive and negetive exammples, so the algorithm holds valid. Case 2: Algorithm halts with failure Required to show if the algorithm fails, there doesn t exist any conjuction that satisfies all the examples. Without loss of generality, let x 1 x 2 x 3... x i be the conjuction obtained from the positive examples. Since, the algorithm fails, there exists some feature vector among the negative examples that is not satisfied by the conjuction. Let that feature vector be F = f 1 f 2 f 3... f i... f n, where f 1 = f 2 = f 3 =... = f i = 1. Suppose that there exists a conjuction C, that satisfies all the negetive and positive examples. Let, C = X 1 X 2, wherex 1 {x 1, x 2, x 3,..., x i } and X 2 {x i+1, x i+2,..., x n }. But X 2 must not be empty set, otherwise C will fail on the feature vector F itself. Hence, it should contain atleast one element from { x i+1, x i+2... x n }. Now this formula should satisfy all the positive examples also. However, each feature element from the set { x i+1, x i+2... x n } is zero in atleast one of the positive examples as our output formula doesnt contain them, i.e. they were dropped during the course of the algorithm as they must have been falsified by some positive example. Hence, C will not satisfy all those positive examples. i.e., C doesn t satisfy all the examples. This brings us to a contradiction. This proves the correctness of our algorithm. 2.5 Extensions Introduction of Negations New vaiables {y 1, y 2,..., y n } are added to the previous set of variables to generate a conjuction, where y i = x i. So, {x 1, x 2,..., x n, y 1, y 2,..., y n } is the new set of variables. Using the same algorithm we can generate a conjuction. Scribe Notes, CSL 758, Advanced Algorithms - 4

5 2-CNF We will introdunce n C 2 variables of the form x i x j, where i j and proceed with this set of variables with the same algorithm. This can be extended to k-cnf form, for any integer k 2. Scribe Notes, CSL 758, Advanced Algorithms - 5