Aims 09s: COMP947 Macine Learning and Data Mining Fundamentals of Concept Learning Marc, 009 Acknowledgement: Material derived from slides for te book Macine Learning, Tom Mitcell, McGraw-Hill, 997 ttp://www-.cs.cmu.edu/~tom/mlbook.tml Tis lecture aims to develop your understanding of representing and searcing ypotesis spaces for concept learning. Following it you sould be able to: define a representation for concepts define a ypotesis space in terms of generality ordering on concepts describe an algoritm to searc a ypotesis space express te framework of version spaces describe an algoritm to searc a ypotesis space using te framework of version spaces explain te role of inductive bias in concept learning COMP947: Marc, 009 Fundamentals of Concept Learning: Slide Overview Concept Learning inferring a Boolean-valued function from training examples of its input and output. Learning from examples General-to-specific ordering over ypoteses Version spaces and candidate elimination algoritm Picking new examples Te need for inductive bias Training Examples for EnjoySport Sky Temp Humid Wind Water Forecst EnjoySpt Sunny Warm Normal Strong Warm Same Yes Sunny Warm Hig Strong Warm Same Yes Rainy Cold Hig Strong Warm Cange No Sunny Warm Hig Strong Cool Cange Yes Wat is te general concept? Note: simple approac assuming no noise, illustrates key concepts COMP947: Marc, 009 Fundamentals of Concept Learning: Slide COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 3
Many possible representations... Representing Hypoteses Here, is a conjunction of constraints on attributes. Given: Te Prototypical Concept Learning Task Instances X: Possible days, eac described by te attributes Eac constraint can be: a specific value (e.g., W ater = W arm) don t care (e.g., W ater =? ) no value allowed (e.g., Water= ) For example, Sky AirTemp Humid Wind Water Forecst Sunny?? Strong? Same Attribute Sky AirTemp Humid Wind Water Forecast Values Sunny, Cloudy, Rainy Warm, Cold Normal, Hig Strong, Weak Warm, Cool Same, Cange COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 4 COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 5 Te Prototypical Concept Learning Task Te inductive learning ypotesis Target function c: EnjoySport : X {0, } Hypoteses H: Conjunctions of literals. E.g.?, Cold, Hig,?,?,?. Training examples D: Positive and negative examples of te target function x, c(x ),... x m, c(x m ) Any ypotesis found to approximate te target function well over a sufficiently large set of training examples will also approximate te target function well over oter unobserved examples. Determine: A ypotesis in H suc tat (x) = c(x) for all x in D (usually called te target ypotesis). COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 6 COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 7
Concept Learning as Searc Question: Wat can be learned? Answer: (only) wat is in te ypotesis space How big is te ypotesis space for EnjoySport? Instance space Sky AirTemp... Forecast = 3 = 96 Concept Learning as Searc Hypotesis space Sky AirTemp... Forecast = 5 4 4 4 4 4 = 50 (semantically distinct only) = + (4 3 3 3 3 3) = 973 any ypotesis wit an constraint covers no instances, ence all are semantically equivalent. Te learning problem searcing a ypotesis space. How? COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 8 COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 9 Instances, Hypoteses, and More-General-Tan Instances X Hypoteses H Specific A generality order on ypoteses Definition: Let j and k be Boolean-valued functions defined over instances X. Ten j is more general tan or equal to k (written j g k ) if and only if x x 3 General ( x X)[( k (x) = ) ( j (x) = )] Intuitively, j is more general tan or equal to k if any instance satisfying k also satisfies j. x = <Sunny, Warm, Hig, Strong, Cool, Same> x = <Sunny, Warm, Hig, Ligt, Warm, Same> = <Sunny,?,?, Strong,?,?> = <Sunny,?,?,?,?,?> = <Sunny,?,?,?, Cool,?> 3 j is (strictly) more general tan k (written j > g k ) if and only if ( j g k ) ( k g j ). j is more specific tan k wen k is more general tan j. COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 0 COMP947: Marc, 009 Fundamentals of Concept Learning: Slide
Te Find-S Algoritm Hypotesis Space Searc by Find-S Instances X Hypoteses H. Initialize to te most specific ypotesis in H. For eac positive training instance x For eac attribute constraint a i in If te constraint a i in is satisfied by x Ten do noting Else replace a i in by te next more general constraint tat is satisfied by x - x 3 x + x+ x+ 4 x = <Sunny Warm Normal Strong Warm Same>, + x = <Sunny Warm Hig Strong Warm Same>, + x 3 = <Rainy Cold Hig Strong Warm Cange>, - x = <Sunny Warm Hig Strong Cool Cange>, + 4 0,3 4 Specific General = <,,,,, > 0 = <Sunny Warm Normal Strong Warm Same> = <Sunny Warm? Strong Warm Same> = <Sunny Warm? Strong Warm Same> 3 = <Sunny Warm? Strong?? > 4 COMP947: Marc, 009 Fundamentals of Concept Learning: Slide COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 3 Find-S - does it work? Complaints about Find-S Assume: a ypotesis c H describes target function c, and training data is error-free. By definition, c is consistent wit all positive training examples and can never cover a negative example. For eac generated by Find-S, c is more general tan or equal to. So can never cover a negative example. Can t tell weter it as learned concept learned ypotesis may not be te only consistent ypotesis Can t tell wen training data inconsistent cannot andle noisy data Picks a maximally specific (wy?) migt require maximally general COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 4 COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 5
Version Spaces Te List-Ten-Eliminate Algoritm A ypotesis is consistent wit a set of training examples D of target concept c if and only if (x) = c(x) for eac training example x, c(x) in D. Consistent(, D) ( x, c(x) D) (x) = c(x). V ersionspace a list containing every ypotesis in H. For eac training example, x, c(x) remove from V ersionspace any ypotesis for wic (x) c(x) 3. Output te list of ypoteses in V ersionspace Te version space, V S H,D, wit respect to ypotesis space H and training examples D, is te subset of ypoteses from H consistent wit all training examples in D. V S H,D { H Consistent(, D)} COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 6 COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 7 Example Version Space Representing Version Spaces S: { <Sunny, Warm,?, Strong,?,?> } Te General boundary, G, of version space V S H,D is te set of its maximally general members G: { <Sunny,?,?,?,?,?>, <?, Warm,?,?,?,?> } Te Specific boundary, S, of version space V S H,D is te set of its maximally specific members Every member of te version space lies between tese boundaries V S H,D = { H ( s S)( g G)(g s)} were x y means x is more general or equal to y COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 8 COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 9
Te Candidate Elimination Algoritm G maximally general ypoteses in H S maximally specific ypoteses in H For eac training example d, do If d is a positive example Remove from G any ypotesis inconsistent wit d For eac ypotesis s in S tat is not consistent wit d Remove s from S Add to S all minimal generalizations of s suc tat. is consistent wit d, and. some member of G is more general tan Remove from S any ypotesis tat is more general tan anoter ypotesis in S If d is a negative example Te Candidate Elimination Algoritm Remove from S any ypotesis inconsistent wit d For eac ypotesis g in G tat is not consistent wit d Remove g from G Add to G all minimal specializations of g suc tat. is consistent wit d, and. some member of S is more specific tan Remove from G any ypotesis tat is less general tan anoter ypotesis in G COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 0 COMP947: Marc, 009 Fundamentals of Concept Learning: Slide Example Trace Example Trace S 0 : {<Ø, Ø, Ø, Ø, Ø, Ø>} S 0 : { <,,,,, > } S : { <Sunny, Warm, Normal, Strong, Warm, Same> } S : { <Sunny, Warm,?, Strong, Warm, Same> } G 0, G, G : { <?,?,?,?,?,?>} G 0 : {<?,?,?,?,?,?>} Training examples:. <Sunny, Warm, Normal, Strong, Warm, Same>, Enjoy Sport = Yes. <Sunny, Warm, Hig, Strong, Warm, Same>, Enjoy Sport = Yes COMP947: Marc, 009 Fundamentals of Concept Learning: Slide COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 3
Example Trace Example Trace S, S 3 : { <Sunny, Warm,?, Strong, Warm, Same> } S 3 : { <Sunny, Warm,?, Strong, Warm, Same> } S 4 : { <Sunny, Warm,?, Strong,?,?>} G 3 : { <Sunny,?,?,?,?,?> <?, Warm,?,?,?,?> <?,?,?,?,?, Same> } G 4: { <Sunny,?,?,?,?,?> <?, Warm,?,?,?,?>} G : { <?,?,?,?,?,?> } G 3 : { <Sunny,?,?,?,?,?> <?, Warm,?,?,?,?> <?,?,?,?,?, Same> } Training Example: 3. <Rainy, Cold, Hig, Strong, Warm, Cange>, EnjoySport=No Training Example: 4.<Sunny, Warm, Hig, Strong, Cool, Cange>, EnjoySport = Yes COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 4 COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 5 Example Trace Wic Training Example Is Best To Coose Next? S 4 : { <Sunny, Warm,?, Strong,?,?>} S: { <Sunny, Warm,?, Strong,?,?> } G 4 : { <Sunny,?,?,?,?,?>, <?, Warm,?,?,?,?>} G: { <Sunny,?,?,?,?,?>, <?, Warm,?,?,?,?> } COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 6 COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 7
Wic Training Example To Coose Next? How Sould New Instances Be Classified? S: { <Sunny, Warm,?, Strong,?,?> } S: { <Sunny, Warm,?, Strong,?,?> } G: { <Sunny,?,?,?,?,?>, <?, Warm,?,?,?,?> } Sunny W arm Normal Ligt W arm Same G: { <Sunny,?,?,?,?,?>, <?, Warm,?,?,?,?> } Sunny Warm Normal Strong Cool Cange Rainy Cold Normal Ligt Warm Same Sunny Warm Normal Ligt Warm Same COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 8 COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 9 How Sould New Instances Be Classified? Wat Justifies tis Inductive Leap? S: { <Sunny, Warm,?, Strong,?,?> } + Sunny W arm Normal Strong Cool Cange + Sunny W arm Normal Ligt W arm Same S : Sunny W arm Normal??? G: { <Sunny,?,?,?,?,?>, <?, Warm,?,?,?,?> } Sunny Warm Normal Strong Cool Cange (6 + /0 ) Rainy Cold Normal Ligt Warm Same (0 + /6 ) Sunny Warm Normal Ligt Warm Same (3 + /3 ) Wy believe we can classify tis unseen instance? Sunny W arm Normal Strong W arm Same COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 30 COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 3
An UNBiased Learner Idea: Coose H tat expresses every teacable concept (i.e. H is te power set of X) Consider H = disjunctions, conjunctions, negations over previous H. E.g. Sunny W arm Normal???????? Cange Wat are S, G in tis case? S G Consider concept learning algoritm L instances X, target concept c Inductive Bias training examples D c = { x, c(x) } let L(x i, D c ) denote te classification assigned to te instance x i by L after training on data D c. Definition: Te inductive bias of L is any minimal set of assertions B suc tat for any target concept c and corresponding training examples D c ( x i X)[(B D c x i ) L(x i, D c )] were A B means A logically entails B COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 3 COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 33 Inductive Systems and Equivalent Deductive Systems Training examples New instance Training examples New instance Inductive system Candidate Elimination Algoritm Using Hypotesis Space H Equivalent deductive system Teorem Prover Classification of new instance, or "don t know" Classification of new instance, or "don t know" Tree Learners wit Different Biases. Rote learner: Store examples, Classify x iff it matces previously observed example.. Version space candidate elimination algoritm 3. Find-S Assertion " H contains te target concept" Inductive bias made explicit COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 34 COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 35
Summary Points. Concept learning as searc troug H. General-to-specific ordering over H 3. Version space candidate elimination algoritm 4. S and G boundaries caracterize learner s uncertainty 5. Learner can generate useful queries 6. Inductive leaps possible only if learner is biased 7. Inductive learners can be modelled by equivalent deductive systems [Suggested reading: Mitcell, Capter ] COMP947: Marc, 009 Fundamentals of Concept Learning: Slide 36