Learning Sets of Rules. Sequential covering algorithms FOIL. Induction as inverse of deduction. Inductive Logic Programming

Size: px

Start display at page:

Download "Learning Sets of Rules. Sequential covering algorithms FOIL. Induction as inverse of deduction. Inductive Logic Programming"

Jasper Singleton
5 years ago
Views:

1 Learning Sets of Rules [Read Ch. 10] [Recommended exercises 10.1, 10.2, 10.5, 10.7, 10.8] Sequential covering algorithms FOIL Induction as inverse of deduction Inductive Logic Programming 229 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

2 Learning Disjunctive Sets of Rules Method 1: Learn decision tree, convert to rules Method 2: Sequential covering algorithm: 1. Learn one rule with high accuracy, anycoverage 2. Remove positive examples covered by this rule 3. Repeat 230 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

3 Sequential Covering Algorithm Sequentialcovering(Target attribute Attributes Examples T hresho Learned rules fg Rule learn-onerule(target attribute Attributes Examples) while performance(rule Examples) > T hreshold, do { Learned rules Learned rules + Rule { Examples Examples ;fexamples correctly classied by Ruleg { Rule learn-onerule(target attribute Attributes Examples) Learned rules performance over Examples return Learned rules sort Learned rules accord to 231 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

4 Learn-One-Rule IF THEN PlayTennis=yes IF Wind=weak THEN PlayTennis=yes IF Wind=strong THEN PlayTennis=no IF Humidity=normal THEN PlayTennis=yes IF Humidity=high THEN PlayTennis=no... IF Humidity=normal Wind=weak THEN PlayTennis=yes IF Humidity=normal Wind=strong THEN PlayTennis=yes IF Humidity=normal Outlook=sunny THEN PlayTennis=yes IF Humidity=normal Outlook=rain THEN PlayTennis=yes lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

5 Learn-One-Rule Pos Neg while Pos,do positive Examples negative Examples Learn a NewRule { N ewrule most general rule possible { NewRuleNeg Neg { while NewRuleNeg, do Add a new literal to specialize NewRule 1. Candidate literals generate candidates 2. Best literal argmax L2Candidate literals P erformance(specializerule(newrule L)) 3. add Best literal to N ewrule preconditions 4. NewRuleNeg subset of NewRuleNeg that satises NewRule preconditions { Learned rules Learned rules + NewRule { Pos Pos ;fmembers of Pos covered by NewRuleg Return Learned rules 233 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

6 Subtleties: Learn One Rule 1. May usebeam search 2. Easily generalizes to multi-valued target functions 3. Choose evaluation function to guide search: Entropy (i.e., information gain) Sample accuracy: n c n where n c = correct rule predictions, n = all predictions m estimate: n c + mp n + m 234 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

7 Variants of Rule Learning Programs Sequential or simultaneous covering of data? General! specic, or specic! general? Generate-and-test, or example-driven? Whether and how to post-prune? What statistical evaluation function? 235 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

8 Learning First Order Rules Why do that? Can learn sets of rules such as Ancestor(x y) Parent(x y) Ancestor(x y) Parent(x z) ^ Ancestor(z y) General purpose programming language Prolog: programs are sets of such rules 236 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

9 First Order Rule for Classifying Web Pages [Slattery, 1997] course(a) has-word(a, instructor), Not has-word(a, good), link-from(a, B), has-word(b, assign), Not link-from(b, C) Train: 31/31, Test: 31/ lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

10 FOIL(Target predicate P redicates Examples) Pos Neg while Pos,do positive Examples negative Examples Learn a NewRule { N ewrule most general rule possible { NewRuleNeg Neg { while NewRuleNeg, do Add a new literal to specialize NewRule 1. Candidate literals generate candidates 2. Best literal argmax L2Candidate literals Foil Gain(L NewRule) 3. add Best literal to N ewrule preconditions 4. NewRuleNeg subset of NewRuleNeg that satises NewRule preconditions { Learned rules Learned rules + NewRule { Pos Pos ;fmembers of Pos covered by NewRuleg Return Learned rules 238 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

11 Specializing Rules in FOIL Learning rule: P (x1 x2 ::: x k ) L1 :::L n Candidate specializations add new literal of form: Q(v1 ::: v r ), where at least one of the v i in the created literal must already exist as a variable in the rule. Equal(x j x k ), where x j and x k are variables already present in the rule The negation of either of the above forms of literals 239 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

12 Information Gain in FOIL F oil Gain(L R) t Where 0 2 p1 p1 + n1 ; log 2 L is the candidate literal to add to rule R p0 =number of positive bindings of R n0 =number of negative bindings of R p1 =number of positive bindings of R + L p0 p0 + n0 n1 =number of negative bindings of R + L t is the number of positive bindings of R also covered by R + L Note p ;log 0 2 p 0 +n 0 is optimal number of bits to indicate the class of a positive binding covered by R 1 C A 240 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

13 Induction as Inverted Deduction Induction is nding h such that (8hx i f(x i )i2d) B ^ h ^ x i ` f(x i ) where x i is ith training instance f(x i ) is the target function value for x i B is other background knowledge So let's design inductive algorithm by inverting operators for automated deduction! 241 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

14 Induction as Inverted Deduction \pairs of people, hu vi such that child of u is v," f(x i ): Child(Bob Sharon) x i : Male(Bob) F emale(sharon) F ather(sharon Bob) B : Parent(u v) F ather(u v) What satises (8hx i f(x i )i2d) B ^ h ^ x i ` f(x i )? h1 : Child(u v) F ather(v u) h2 : Child(u v) Parent(v u) 242 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

15 Induction is, in fact, the inverse operation of deduction, and cannot be conceived to exist without the corresponding operation, so that the question of relative importance cannot arise. Who thinks of asking whether addition or subtraction is the more important process in arithmetic? But at the same time much dierence in diculty may exist between a direct and inverse operation ::: it must be allowed that inductive investigations are of a far higher degree of diculty and complexity than any questions of deduction:::: (Jevons 1874) 243 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

16 Induction as Inverted Deduction We have mechanical deductive operators F (A B) =C, where A ^ B ` C need inductive operators O(B D) =h where (8hx i f(x i )i2d)(b^h^x i ) ` f(x i ) 244 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

17 Induction as Inverted Deduction Positives: Subsumes earlier idea of nding h that \ts" training data Domain theory B helps dene meaning of \t" the data B ^ h ^ x i ` f(x i ) Suggests algorithms that search H guided by B 245 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

18 Induction as Inverted Deduction Negatives: Doesn't allow for noisy data. Consider (8hx i f(x i )i2d) (B ^ h ^ x i ) ` f(x i ) First order logic gives a huge hypothesis space H! overtting...! intractability of calculating all acceptable h's 246 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

19 Deduction: Resolution Rule P _ L :L _ R P _ R 1. Given initial clauses C1 and C2, nd a literal L from clause C1 such that :L occurs in clause C2 2. Form the resolvent C by including all literals from C1 and C2, except for L and :L. More precisely, the set of literals occurring in the conclusion C is C =(C1 ;flg) [ (C2 ;f:lg) where [ denotes set union, and \;" denotes set dierence. 247 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

20 Inverting Resolution C : 2 KnowMaterial V Study C : 2 KnowMaterial V Study C : 1 PassExam V KnowMaterial C : PassExam V KnowMaterial 1 C: PassExam V Study C: PassExam V Study 248 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

21 Inverted Resolution (Propositional) 1. Given initial clauses C1 and C, nd a literal L that occurs in clause C1, but not in clause C. 2. Form the second clause C2 by including the following literals C2 =(C ; (C1 ;flg)) [f:lg 249 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

22 First order resolution First order resolution: 1. Find a literal L1 from clause C1, literall2 from clause C2, and substitution such that L1 = :L2 2. Form the resolvent C by including all literals from C1 and C2, except for L1 and :L2. More precisely, the set of literals occurring in the conclusion C is C =(C1 ;fl1g) [ (C2 ;fl2g) 250 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

23 Inverting First order resolution C2 =(C ; (C1 ;fl1g)1) ;1 2 [f:l11 ;1 2 g 251 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

24 Cigol Father ( Tom, Bob ) GrandChild ( y,x ) V Father ( x,z ) V Father ( z,y) { Bob/y, Tom/z} Father ( Shannon, Tom) GrandChild ( Bob,x ) V Father ( x,tom) { Shannon/x} GrandChild ( Bob, Shannon) 252 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

25 Progol Progol: Reduce comb explosion by generating the most specic acceptable h 1. User species H by stating predicates, functions, and forms of arguments allowed for each 2. Progol uses sequential covering algorithm. For each hx i f(x i )i Find most specic hypothesis h i s.t. B ^ h i ^ x i ` f(x i ) { actually, considers only k-step entailment 3. Conduct general-to-specic search bounded by specic hypothesis h i,choosing hypothesis with minimum description length 253 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

Introduction. Decision Tree Learning. Outline. Decision Tree 9/7/2017. Decision Tree Definition

Introduction. Decision Tree Learning. Outline. Decision Tree 9/7/2017. Decision Tree Definition Introduction Decision Tree Learning Practical methods for inductive inference Approximating discrete-valued functions Robust to noisy data and capable of learning disjunctive expression ID3 earch a completely