CSE4334/5334 Data Mining Association Rule Mining. Chengkai Li University of Texas at Arlington Fall 2017

Size: px

Start display at page:

Download "CSE4334/5334 Data Mining Association Rule Mining. Chengkai Li University of Texas at Arlington Fall 2017"

Pamela Chambers
6 years ago
Views:

1 CSE4334/5334 Data Mining Assciatin Rule Mining Chengkai Li University f Texas at Arlingtn Fall 27

2 Assciatin Rule Mining Given a set f transactins, find rules that will predict the ccurrence f an item based n the ccurrences f ther items in the transactin Market-Basket transactins TID Items Bread, Milk 2 Bread, Diaper, Beer, Eggs 3 Milk, Diaper, Beer, Cke 4 Bread, Milk, Diaper, Beer 5 Bread, Milk, Diaper, Cke Example f Assciatin Rules {Diaper} {Beer}, {Milk, Bread} {Eggs,Cke}, {Beer, Bread} {Milk}, Implicatin means c-ccurrence, nt causality!

3 Definitin: Frequent Itemset Itemset A cllectin f ne r mre items Example: {Milk, Bread, Diaper} k-itemset An itemset that cntains k items Supprt cunt (s) Frequency f ccurrence f an itemset E.g. s({milk, Bread,Diaper}) = 2 Supprt Fractin f transactins that cntain an itemset E.g. s({milk, Bread, Diaper}) = 2/5 Frequent Itemset An itemset whse supprt is greater than r equal t a minsup threshld TID Items Bread, Milk 2 Bread, Diaper, Beer, Eggs 3 Milk, Diaper, Beer, Cke 4 Bread, Milk, Diaper, Beer 5 Bread, Milk, Diaper, Cke

4 Definitin: Assciatin Rule Assciatin Rule An implicatin expressin f the frm X Y, where X and Y are itemsets Example: {Milk, Diaper} {Beer} Rule Evaluatin Metrics Supprt (s) u Fractin f transactins that cntain bth X and Y Cnfidence (c) u Measures hw ften items in Y appear in transactins that cntain X TID Example: Items Bread, Milk 2 Bread, Diaper, Beer, Eggs 3 Milk, Diaper, Beer, Cke 4 Bread, Milk, Diaper, Beer 5 Bread, Milk, Diaper, Cke { Milk, Diaper} Þ (Milk, Diaper,Beer) s = s = T s (Milk,Diaper,Beer) c = = s (Milk,Diaper) = Beer =.4.67

5 Assciatin Rule Mining Task Given a set f transactins T, the gal f assciatin rule mining is t find all rules having supprt minsup threshld cnfidence mincnf threshld Brute-frce apprach: List all pssible assciatin rules Cmpute the supprt and cnfidence fr each rule Prune rules that fail the minsup and mincnf threshlds Þ Cmputatinally prhibitive!

6 Mining Assciatin Rules TID Items Bread, Milk 2 Bread, Diaper, Beer, Eggs 3 Milk, Diaper, Beer, Cke 4 Bread, Milk, Diaper, Beer 5 Bread, Milk, Diaper, Cke Example f Rules: {Milk,Diaper} {Beer} (s=.4, c=.67) {Milk,Beer} {Diaper} (s=.4, c=.) {Diaper,Beer} {Milk} (s=.4, c=.67) {Beer} {Milk,Diaper} (s=.4, c=.67) {Diaper} {Milk,Beer} (s=.4, c=.5) {Milk} {Diaper,Beer} (s=.4, c=.5) Observatins: All the abve rules are binary partitins f the same itemset: {Milk, Diaper, Beer} Rules riginating frm the same itemset have identical supprt but can have different cnfidence Thus, we may decuple the supprt and cnfidence requirements

7 Mining Assciatin Rules Tw-step apprach:. Frequent Itemset Generatin Generate all itemsets whse supprt ³ minsup 2. Rule Generatin Generate high cnfidence rules frm each frequent itemset, where each rule is a binary partitining f a frequent itemset Frequent itemset generatin is still cmputatinally expensive

8 Frequent Itemset Generatin null A B C D E AB AC AD AE BC BD BE CD CE DE ABC ABD ABE ACD ACE ADE BCD BCE BDE CDE ABCD ABCE ABDE ACDE BCDE ABCDE Given d items, there are 2 d pssible candidate itemsets

9 Frequent Itemset Generatin Brute-frce apprach: Each itemset in the lattice is a candidate frequent itemset Cunt the supprt f each candidate by scanning the database N Transactins TID Items Bread, Milk 2 Bread, Diaper, Beer, Eggs 3 Milk, Diaper, Beer, Cke 4 Bread, Milk, Diaper, Beer 5 Bread, Milk, Diaper, Cke w List f Candidates Match each transactin against every candidate Cmplexity ~ O(NMw) => Expensive since M = 2 d!!! M

10 Cmputatinal Cmplexity Given d unique items: Ttal number f itemsets = 2 d Ttal number f pssible assciatin rules: = ú û ù ê ë é ø ö ç è æ - ø ö ç è æ = + - = - = å å d d d k k d j j k d k d R If d=6, R = 62 rules

11 Frequent Itemset Generatin Strategies Reduce the number f candidates (M) Cmplete search: M=2 d Use pruning techniques t reduce M Reduce the number f transactins (N) Reduce size f N as the size f itemset increases Used by DHP and vertical-based mining algrithms Reduce the number f cmparisns (NM) Use efficient data structures t stre the candidates r transactins N need t match every candidate against every transactin

12 Reducing Number f Candidates Apriri principle: If an itemset is frequent, then all f its subsets must als be frequent Apriri principle hlds due t the fllwing prperty f the supprt measure: " X, Y : ( X Í Y ) Þ s( X ) ³ s( Y Supprt f an itemset never exceeds the supprt f its subsets This is knwn as the anti-mntne prperty f supprt )

13 Illustrating Apriri Principle null A B C D E AB AC AD AE BC BD BE CD CE DE Fund t be Infrequent ABC ABD ABE ACD ACE ADE BCD BCE BDE CDE ABCD ABCE ABDE ACDE BCDE Pruned supersets ABCDE

14 Illustrating Apriri Principle Item Cunt Bread 4 Cke 2 Milk 4 Beer 3 Diaper 4 Eggs Minimum Supprt = 3 Items (-itemsets) Itemset Cunt {Bread,Milk} 3 {Bread,Beer} 2 {Bread,Diaper} 3 {Milk,Beer} 2 {Milk,Diaper} 3 {Beer,Diaper} 3 Pairs (2-itemsets) (N need t generate candidates invlving Cke r Eggs) Triplets (3-itemsets) If every subset is cnsidered, 6 C + 6 C C 3 = 4 With supprt-based pruning, = 3 Itemset Cunt {Bread,Milk,Diaper} 3

15 Apriri Algrithm Methd: Let k= Generate frequent itemsets f length Repeat until n new frequent itemsets are identified Generate length (k+) candidate itemsets frm length k frequent itemsets Prune candidate itemsets cntaining subsets f length k that are infrequent Cunt the supprt f each candidate by scanning the DB Eliminate candidates that are infrequent, leaving nly thse that are frequent

16 Anther Example Database TDB Tid Items A, B, D 2 A, C, E 3 A, B, C, E 4 C, E Sup min = 2 st scan Itemset sup {A} 3 L C {B} 2 {C} 3 {D} {E} 3 C 2 Itemset sup C 2 {A, B} 2 L 2 Itemset sup 2 nd scan {A, C} 2 {A, B} 2 {A, E} 2 {A, C} 2 {B, C} {A, E} 2 {B, E} {C, E} 3 {C, E} 3 Itemset sup {A} 3 {B} 2 {C} 3 {E} 3 Itemset {A, B} {A, C} {A, E} {B, C} {B, E} {C, E} C 3 Itemset 3 rd scan L 3 {A, C, E} Itemset sup {A, C, E} 2 6

17 Apriri Algrithm Let k= Generate frequent itemsets f length Repeat until n new frequent itemsets are identified. Generate candidate (k+)-itemsets frm frequent k- itemsets 2. Prune candidate (k+)-itemsets cntaining sme infrequent k-itemset 3. Cunt the supprt f each candidate by scanning the DB 4. Eliminate infrequent candidates, leaving nly thse that are frequent 7

18 . Generate Candidate (k+) itemsets Sup min = 2 Input: frequent k-itemsets L k L 2 Itemset sup {A, B} 2 {A, C} 2 {A, E} 2 {C, E} 3 Output: frequent (k+)-itemsets L k+ Prcedure:. Candidate generatin, by self-jin L k *L k q Fr each pair f P={p, p 2,, p k }ÎL k, q={q, q 2,, q k } ÎL k, C 3 Itemset {A, B, C} {A, B, E} {A, C, E} if p =q,, p k- =q k-, p k < q k, add {p,, p k-, p k, q k} int C k+ Example: L 2 ={AB, AC, AE, CE} n AB and AC => ABC n AB and AE => ABE n AC and AE => ACE 8

19 2. Prune Candidates L 2 Sup min = 2 Itemset sup {A, B} 2 {A, C} 2 {A, E} 2 {C, E} 3 Input: frequent k-itemsets L k Output: frequent (k+)-itemsets L k+ Prcedure:. Candidate generatin, by self-jin L k *L k q Fr each pair f P={p, p 2,, p k }ÎL 2, q={q, q 2,, q k } ÎL 2, if p =q,, p k- =q k-, p k < q k, add {p,, p k-, p k, q k} int C k+ C 3 Itemset {A, B, C} {A, B, E} {A, C, E} 2. Prune candidates that cntain infrequent k-itemsets Example: L 2 ={AB, AC, AE, CE} n n n AB and AC => ABC, pruned because BC is nt frequent AB and AE => ABE, pruned because BE is nt frequent AC and AE => ACE 9

20 3. Cunt supprt f candidates and 4. Eliminate infrequent candidates Sup min = 2 Input: frequent k-itemsets L k L 2 Itemset sup {A, B} 2 {A, C} 2 {A, E} 2 {C, E} 3 Output: frequent (k+)-itemsets L k+ Prcedure:. Candidate generatin, by self-jin L k *L k q Fr each pair f P={p, p 2,, p k }ÎL 2, q={q, q 2,, q k } ÎL 2, C 3 3 rd scan Itemset {A, C, E} if p =q,, p k- =q k-, p k < q k, add {p,, p k-, p k, q k} int C k+ 2. Prune candidates that cntain infrequent k-itemsets L 3 Itemset sup {A, C, E} 2 3. Cunt the supprt f each candidate by scanning the DB 4. Eliminate infrequent candidates 2

21 Reducing Number f Cmparisns Candidate cunting: Scan the database f transactins t determine the supprt f each candidate itemset T reduce the number f cmparisns, stre the candidates in a hash structure Instead f matching each transactin against every candidate, match it against candidates cntained in the hashed buckets Transactins Hash Structure N TID Items Bread, Milk 2 Bread, Diaper, Beer, Eggs 3 Milk, Diaper, Beer, Cke 4 Bread, Milk, Diaper, Beer 5 Bread, Milk, Diaper, Cke k Buckets

22 Generate Hash Tree Suppse yu have 5 candidate itemsets f length 3: { 4 5}, { 2 4}, {4 5 7}, { 2 5}, {4 5 8}, { 5 9}, { 3 6}, {2 3 4}, {5 6 7}, {3 4 5}, {3 5 6}, {3 5 7}, {6 8 9}, {3 6 7}, {3 6 8} Yu need: Hash functin Max leaf size: max number f itemsets stred in a leaf nde (if number f candidate itemsets exceeds max leaf size, split the nde) Hash functin 3,6,9,4,7 2,5,

23 Assciatin Rule Discvery: Hash tree Hash Functin Candidate Hash Tree,4,7 3,6,9 2,5,8 Hash n, 4 r

24 Assciatin Rule Discvery: Hash tree Hash Functin Candidate Hash Tree,4,7 3,6,9 2,5,8 Hash n 2, 5 r

25 Assciatin Rule Discvery: Hash tree Hash Functin Candidate Hash Tree,4,7 3,6,9 2,5,8 Hash n 3, 6 r

26 Subset Operatin Given a transactin t, what are the pssible subsets f size 3? Transactin, t Level Level Level 3 Subsets f 3 items

27 Subset Operatin Using Hash Tree transactin Hash Functin ,4,7 3,6, ,5,

28 Subset Operatin Using Hash Tree transactin Hash Functin ,4,7 2,5,8 3,6,

29 Subset Operatin Using Hash Tree transactin Hash Functin ,4,7 2,5,8 3,6, Match transactin against ut f 5 candidates

30 Factrs Affecting Cmplexity Chice f minimum supprt threshld lwering supprt threshld results in mre frequent itemsets this may increase number f candidates and max length f frequent itemsets Dimensinality (number f items) f the data set mre space is needed t stre supprt cunt f each item if number f frequent items als increases, bth cmputatin and I/O csts may als increase Size f database since Apriri makes multiple passes, run time f algrithm may increase with number f transactins Average transactin width transactin width increases with denser data sets This may increase max length f frequent itemsets and traversals f hash tree (number f subsets in a transactin increases with its width)

31 Rule Generatin Given a frequent itemset L, find all nn-empty subsets f Ì L such that f L f satisfies the minimum cnfidence requirement If {A,B,C,D} is a frequent itemset, candidate rules: ABC D, ABD C, ACD B, BCD A, A BCD, B ACD, C ABD, D ABC AB CD, AC BD, AD BC, BC AD, BD AC, CD AB, If L = k, then there are 2 k 2 candidate assciatin rules (ignring L Æ and Æ L)

32 Rule Generatin Hw t efficiently generate rules frm frequent itemsets? In general, cnfidence des nt have an anti-mntne prperty c(abc D) can be larger r smaller than c(ab D) But cnfidence f rules generated frm the same itemset has an anti-mntne prperty e.g., L = {A,B,C,D}: c(abc D) ³ c(ab CD) ³ c(a BCD) Cnfidence is anti-mntne w.r.t. number f items n the RHS f the rule

33 Rule Generatin fr Apriri Algrithm Lattice f rules Lw Cnfidence Rule ABCD=>{ } BCD=>A ACD=>B ABD=>C ABC=>D CD=>AB BD=>AC BC=>AD AD=>BC AC=>BD AB=>CD Pruned Rules D=>ABC C=>ABD B=>ACD A=>BCD

34 Rule Generatin fr Apriri Algrithm Candidate rule is generated by merging tw rules that share the same prefix in the rule cnsequent jin(cd=>ab,bd=>ac) wuld prduce the candidate rule D => ABC CD=>AB BD=>AC Prune rule D=>ABC if its subset AD=>BC des nt have high cnfidence D=>ABC

35 Pattern Evaluatin Assciatin rule algrithms tend t prduce t many rules many f them are uninteresting r redundant Redundant if {A,B,C} {D} and {A,B} {D} have same supprt & cnfidence Interestingness measures can be used t prune/rank the derived patterns In the riginal frmulatin f assciatin rules, supprt & cnfidence are the nly measures used

36 Cmputing Interestingness Measure Given a rule X Y, infrmatin needed t cmpute rule interestingness can be btained frm a cntingency table Cntingency table fr X Y Y Y X f f f + X f f f + f + f + T f : supprt f X and Y f : supprt f X and Y f : supprt f X and Y f : supprt f X and Y Used t define varius measures supprt, cnfidence, lift, Gini, J-measure, etc.

37 Drawback f Cnfidence Cffee Cffee Tea Tea Assciatin Rule: Tea Cffee Cnfidence= P(Cffee Tea) =.75 but P(Cffee) =.9 Þ Althugh cnfidence is high, rule is misleading Þ P(Cffee Tea) =.9375

38 Statistical Independence Ppulatin f students 6 students knw hw t swim (S) 7 students knw hw t bike (B) 42 students knw hw t swim and bike (S,B) P(SÙB) = 42/ =.42 P(S) P(B) =.6.7 =.42 P(SÙB) = P(S) P(B) => Statistical independence P(SÙB) > P(S) P(B) => Psitively crrelated P(SÙB) < P(S) P(B) => Negatively crrelated

39 Statistical-based Measures Measures that take int accunt statistical dependence )] ( )[ ( )] ( )[ ( ) ( ) ( ), ( ) ( ) ( ), ( ) ( ) ( ), ( ) ( ) ( Y P Y P X P X P Y P X P Y X P cefficient Y P X P Y X P PS Y P X P Y X P Interest Y P X Y P Lift = - - = = = f

40 Example: Lift/Interest Cffee Cffee Tea Tea Assciatin Rule: Tea Cffee Cnfidence= P(Cffee Tea) =.75 but P(Cffee) =.9 Þ Lift =.75/.9=.8333 (<, therefre is negatively assciated)

41 Drawback f Lift & Interest Y Y X X Y Y X 9 9 X 9. Lift = =.9 Lift = =. (.)(.) (.9)(.9) Statistical independence: If P(X,Y)=P(X)P(Y) => Lift =

42 There are lts f measures prpsed in the literature Sme measures are gd fr certain applicatins, but nt fr thers What criteria shuld we use t determine whether a measure is gd r bad? What abut Apriristyle supprt based pruning? Hw des it affect these measures?

43 Prperties f A Gd Measure Piatetsky-Shapir: 3 prperties a gd measure M must satisfy: M(A,B) = if A and B are statistically independent M(A,B) increase mntnically with P(A,B) when P(A) and P(B) remain unchanged M(A,B) decreases mntnically with P(A) [r P(B)] when P(A,B) and P(B) [r P(A)] remain unchanged

44 Cmparing Different Measures examples f cntingency tables: Rankings f cntingency tables using varius measures: Example f f f f E E E E E E E E E E

45 Prperty under Variable Permutatin B B A p q A r s A A B p r B q s Des M(A,B) = M(B,A)? Symmetric measures: supprt, lift, cllective strength, csine, Jaccard, etc Asymmetric measures: cnfidence, cnvictin, Laplace, J-measure, etc

46 Prperty under Rw/Clumn Scaling Grade-Gender Example (Msteller, 968): Male Female High Lw Male Female High Lw x x Msteller: Underlying assciatin shuld be independent f the relative number f male and female students in the samples

47 Prperty under Inversin Operatin A B C D (a) (b) (c) E F Transactin Transactin N.....

48 Example: f-cefficient f-cefficient is analgus t crrelatin cefficient fr cntinuus variables Y Y X 6 7 X Y Y X 2 3 X f = = f = f Cefficient is the same fr bth tables =

49 Prperty under Null Additin B B A p q A r s B B A p q A r s + k Invariant measures: supprt, csine, Jaccard, etc Nn-invariant measures: crrelatin, Gini, mutual infrmatin, dds rati, etc

50 Different Measures have Different Prperties Sym bl Measure Range P P2 P3 O O2 O3 O3' O4 F Crrelatin - Yes Yes Yes Yes N Yes Yes N l Lambda Yes N N Yes N N* Yes N a Odds rati Yes* Yes Yes Yes Yes Yes* Yes N Q Yule's Q - Yes Yes Yes Yes Yes Yes Yes N Y Yule's Y - Yes Yes Yes Yes Yes Yes Yes N k Chen's - Yes Yes Yes Yes N N Yes N M Mutual Infrmatin Yes Yes Yes Yes N N* Yes N J J-Measure Yes N N N N N N N G Gini Index Yes N N N N N* Yes N s Supprt N Yes N Yes N N N N c Cnfidence N Yes N Yes N N N Yes L Laplace N Yes N Yes N N N N V Cnvictin.5 N Yes N Yes** N N Yes N I Interest Yes* Yes Yes Yes N N N N IS IS (csine).. N Yes Yes Yes N N N Yes PS Piatetsky-Shapir's Yes Yes Yes Yes N Yes Yes N F Certainty factr - Yes Yes Yes N N N Yes N AV Added value.5 Yes Yes Yes N N N N N S Cllective strength N Yes Yes Yes N Yes* Yes N z Jaccard.. N Yes Yes Yes N N N Yes æ 2 ö K Klsgen's ç 2 æ ö - ç2-3 -!! Yes Yes Yes N N N N N 3 è øè 3 ø 3 3

DATA MINING - 1DL360

DATA MINING - 1DL360 DATA MINING - 1DL36 Fall 212" An introductory class in data mining http://www.it.uu.se/edu/course/homepage/infoutv/ht12 Kjell Orsborn Uppsala Database Laboratory Department of Information Technology, Uppsala