COMP9318: Data Warehousing and Data Mining

Transcription

1 COMP9318: Data Warehousig ad Data Miig L6: Associatio Rule Miig COMP9318: Data Warehousig ad Data Miig 1

2 Problem defiitio ad prelimiaries COMP9318: Data Warehousig ad Data Miig 2

3 What Is Associatio Miig? Associatio rule miig: Fidig frequet patters, associatios, correlatios, or causal structures amog sets of items or objects i trasactio databases, relatioal databases, ad other iformatio repositories. Frequet patter: patter (set of items, sequece, etc.) that occurs frequetly i a database [AIS93] Motivatio: fidig regularities i data What products were ofte purchased together? Beer ad diapers?! What are the subsequet purchases after buyig a PC? What kids of DNA are sesitive to this ew drug? Ca we automatically classify web documets? COMP9318: Data Warehousig ad Data Miig 3

4 Why Is Frequet Patter or Assoiciatio Miig a Essetial Task i Data Miig? Foudatio for may essetial data miig tasks Associatio, correlatio, causality Sequetial patters, temporal or cyclic associatio, partial periodicity, spatial ad multimedia associatio Associative classificatio, cluster aalysis, iceberg cube, fascicles (sematic data compressio) Broad applicatios Basket data aalysis, cross-marketig, catalog desig, sale campaig aalysis Web log (click stream) aalysis, DNA sequece aalysis, etc. c.f., google s spellig suggestio COMP9318: Data Warehousig ad Data Miig

5 Basic Cocepts: Frequet Patters ad Associatio Rules Itemset X={x 1,, x k } Trasactio-id Customer buys both Items bought 10 { A, B, C } 20 { A, C } 30 { A, D } 40 { B, E, F } Customer buys diaper Shorthad: x 1 x 2 x k Fid all the rules XàY with mi cofidece ad support support, s, probability that a trasactio cotais XÈY cofidece, c, coditioal probability that a trasactio havig X also cotais Y. Customer buys beer Let mi_support = 50%, mi_cof = 70%: sup(ac) = 2 A è C (50%, 66.7%) C è A (50%, 100%) frequet itemset associatio rule COMP9318: Data Warehousig ad Data Miig 5

6 Miig Associatio Rules a Example Trasactio-id Items bought 10 A, B, C 20 A, C 30 A, D 40 B, E, F Mi. support 50% Mi. cofidece 50% Frequet patter Support {A} 75% {B} 50% For rule A è C: support = support({a} {C}) = 50% {C} 50% {A, C} 50% cofidece = support({a} {C})/support({A}) = 66.6% major computatio challege: calculate the support of itemsets ç The frequet itemset miig problem COMP9318: Data Warehousig ad Data Miig 6

7 Algorithms for scalable miig of (sigle-dimesioal Boolea) associatio rules i trasactioal databases COMP9318: Data Warehousig ad Data Miig 7

8 Associatio Rule Miig Algorithms Naïve algorithm Eumerate all possible itemsets ad check their support agaist mi_sup Geerate all associatio rules ad check their cofidece agaist mi_cof The Apriori property Apriori Algorithm FP-growth Algorithm Cadidate Geeratio & Verificatio COMP9318: Data Warehousig ad Data Miig 8

9 All Cadidate Itemsets for {A, B, C, D, E} ull A B C D E AB AC AD AE BC BD BE CD CE DE ABC ABD ABE ACD ACE ADE BCD BCE BDE CDE ABCD ABCE ABDE ACDE BCDE ABCDE COMP9318: Data Warehousig ad Data Miig 9

10 Apriori Property A frequet (used to be called large) itemset is a itemset whose support is mi_sup. Apriori property (dowward closure): ay subsets of a frequet itemset are also frequet itemsets Aka the ati-mootoe property of support ABC ABD ACD BCD AB AC AD BC BD CD A B C D ay supersets of a ifrequet itemset are also ifrequet itemsets COMP9318: Data Warehousig ad Data Miig 10

11 Illustratig Apriori Priciple Q: How to desig a algorithm to improve the aïve algorithm? ull A B C D E AB AC AD AE BC BD BE CD CE DE Foud to be Ifrequet ABC ABD ABE ACD ACE ADE BCD BCE BDE CDE ABCD ABCE ABDE ACDE BCDE Prued supersets ABCDE COMP9318: Data Warehousig ad Data Miig 11

12 Apriori: A Cadidate Geeratio-ad-test Approach Apriori pruig priciple: If there is ay itemset which is ifrequet, its superset should ot be geerated/tested! Algorithm [Agrawal & Srikat 1994] 1. C k ç Perform level-wise cadidate geeratio (from sigleto itemsets) 2. L k ç Verify C k agaist L k 3. C k+1 ç geerated from L k 4. Goto 2 if C k+1 is ot empty COMP9318: Data Warehousig ad Data Miig 12

13 The Apriori Algorithm Pseudo-code: C k : Cadidate itemset of size k L k : frequet itemset of size k L 1 = {frequet items}; for (k = 1; L k!= ; k++) do begi C k+1 = cadidates geerated from L k ; for each trasactio t i database do begi icremet the cout of all cadidates i C k+1 that are cotaied i t ed L k+1 = cadidates i C k+1 with mi_support ed retur k L k ; COMP9318: Data Warehousig ad Data Miig 13

14 The Apriori Algorithm A Example Database TDB Tid Items 10 A, C, D 20 B, C, E 30 A, B, C, E 40 B, E 1 st sca Itemset L C 1 1 {B} 3 C 2 C 2 {A, B} 1 L 2 Itemset sup 2 d sca {A, C} 2 {B, C} 2 {B, E} 3 {C, E} 2 C Itemset 3 3 rd sca L 3 {B, C, E} Itemset sup {A} 2 {C} 3 {D} 1 {E} 3 sup {A, C} 2 {A, E} 1 {B, C} 2 {B, E} 3 {C, E} 2 Itemset sup {B, C, E} 2 Itemset sup {A} 2 {B} 3 {C} 3 {E} 3 Itemset {A, B} {A, C} {A, E} {B, C} {B, E} {C, E} misup = 50% COMP9318: Data Warehousig ad Data Miig 14

15 Importat Details of Apriori 1. How to geerate cadidates? Step 1: self-joiig L k (what s the joi coditio? why?) Step 2: pruig 2. How to cout supports of cadidates? Example of Cadidate-geeratio L 3 ={abc, abd, acd, ace, bcd} Self-joiig: L 3 *L 3 abcd from abc ad abd acde from acd ad ace Pruig: acde is removed because ade is ot i L 3 C 4 ={abcd} COMP9318: Data Warehousig ad Data Miig 15

16 Geeratig Cadidates i SQL Suppose the items i L k-1 are listed i a order Step 1: self-joiig L k-1 isert ito C k select p.item 1, p.item 2,, p.item k-1, q.item k-1 from L k-1 p, L k-1 q where p.item 1 =q.item 1,, p.item k-2 =q.item k-2, p.item k-1 < q.item k-1 Step 2: pruig forall itemsets c i C k do forall (k-1)-subsets s of c do if (s is ot i L k-1 ) the delete c from C k COMP9318: Data Warehousig ad Data Miig 16

17 Derive rules from frequet itemsets Frequet itemsets!= associatio rules Oe more step is required to fid associatio rules For each frequet itemset X, For each proper oempty subset A of X, Let B = X - A A à B is a associatio rule if Cofidece (A à B) mi_cof, where support (A à B) = support (AB), ad cofidece (A à B) = support (AB) / support (A) COMP9318: Data Warehousig ad Data Miig 17

18 Example derivig rules from frequet itemsets Suppose 234 is frequet, with supp=50% Proper oempty subsets: 23, 24, 34, 2, 3, 4, with supp=50%, 50%, 75%, 75%, 75%, 75% respectively These geerate these associatio rules: 23 => 4, cofidece=100% 24 => 3, cofidece=100% 34 => 2, cofidece=67% 2 => 34, cofidece=67% 3 => 24, cofidece=67% 4 => 23, cofidece=67% All rules have support = 50% = (N* 50%)/(N*75%) Q: is there ay optimizatio (e.g., pruig) for this step? COMP9318: Data Warehousig ad Data Miig 18

19 Derivig rules To recap, i order to obtai A à B, we eed to have Support(AB) ad Support(A) This step is ot as time-cosumig as frequet itemsets geeratio Why? It s also easy to speedup usig techiques such as parallel processig. How? Do we really eed cadidate geeratio for derivig associatio rules? Frequet-Patter Growth (FP-Tree) COMP9318: Data Warehousig ad Data Miig 19

20 Bottleeck of Frequet-patter Miig Multiple database scas are costly Miig log patters eeds may passes of scaig ad geerates lots of cadidates To fid frequet itemset i 1 i 2 i 100 # of scas: 100 # of Cadidates: Bottleeck: cadidate-geeratio-ad-test = Ca we avoid cadidate geeratio altogether? COMP9318: Data Warehousig ad Data Miig 20

21 FP-growth COMP9318: Data Warehousig ad Data Miig 21

22 No Pai, No Gai Java Lisp Scheme Pytho Ruby Alice X X Bob X X Charlie X X X Dora X X misup = 1 Apriori: L1 = {J, L, S, P, R} C2 = all the ( 5 2) combiatios Most of C2 do ot cotribute to the result There is o way to tell because

23 No Pai, No Gai Java Lisp Scheme Pytho Ruby Alice X X Bob X X Charlie X X X Dora X X Ideas: Keep the support set for each frequet itemset DFS misup = 1 J è JL? J è??? Oly eed to look at support set for J {A, C} J ɸ

24 No Pai, No Gai Java Lisp Scheme Pytho Ruby Alice X X Bob X X Charlie X X X Dora X X Ideas: Keep the support set for each frequet itemset DFS misup = 1 {C} JP {C} JR JPR {A,C} {A, C} J ɸ

25 Notatios ad Ivariats CodiditoalDB: DB p = {t DB t cotais itemset p} DB = DB (i.e., coditioed o othig) Shorthad: DB px = DB (p x) SupportSet(p x, DB) = SupportSet(x, DB p) {x x mod 6 = 0 x [100] } = {x x mod 3 = 0 x eve([100]) } A FP-tree is equivalet to a DB p Oe ca be coverted to aother Next, we illustrate the alg usig coditioaldb 25

26 FP-tree Essetial Idea /1 Recursive algorithm agai! FreqItemsets(DB p): easy task, as oly items (ot itemsets) are eeded all frequet itemsets i DB p belog to oe of the followig categories: X = FidLocallyFrequetItems(DB p) patters ~ x i p output { (x p) x X } patters ~ px 1 Foreach x i X DB* px = GetCoditioalDB + (DB* p, x) obtaied via recursio patters ~ px 2 patters ~ px i patters ~ px FreqItemsets(DB* px)

27 No Pai, No Gai DB J Java Lisp Scheme Pytho Ruby Alice X X Charlie X X X misup = 1 FreqItemsets(DB J): {P, R} ç FidLocallyFrequetItems(DB J) Output {JP, JR} Get DB* JP; FreqItemsets(DB* JP) Get DB* JR; FreqItemsets(DB* JR) // Guarateed o other frequet itemset i DB J

28 FP-tree Essetial Idea /2 FreqItemsets(DB p): If boudary coditio, the X = FidLocallyFrequetItems(DB p) [optioal] DB* p = PrueDB(DB p, X) output { (x p) x X } Foreach x i X DB* px = GetCoditioalDB + (DB* p, x) Also output each item i X (appeded with the coditioal patter) Remove items ot i X; potetially reduce # of trasactios ( or dup). Improves the efficiecy. [optioal] if DB* px is degeerated, the powerset(db* px) FreqItemsets(DB* px) Also gets rid of items already processed before x è avoid duplicates

29 Lv 1 Recursio misup = 3 Grayed items are for illustratio purpose oly. F C A M P C B P DB* P F C A M P F C A D G I M P A B C F L M O B F H J O W B C K S P A F C E L P M N DB F C A M P F C A B M F B C B P F C A M P DB* DB* M (sas P) DB* B (sas MP) DB* A (sas BMP) DB* C (sas ABMP) DB* F (sas CABMP) X = {F, C, A, B, M, P} Output: F, C, A, B, M, P F C A F C A F C A

30 Lv 2 Recursio o DB* P misup = 3 Which is actually FullDB* CP F C A M P C B P F C A M P C C C DB* C C C C DB X = {C} Output: CP DB* Cotext = Lv 3 recursio o DB* CP: DB has oly empty sets or X = {} è immediately returs

31 Lv 2 Recursio o DB* A (sas ) misup = 3 Which is actually FullDB* CA Further recursio (output: FCA) F C A F C A F C A DB F C F C F C DB* DB* C DB* F FC FC FC F F F X = {F, C} Output: FA, CA boudary case

32 Differet Example: Lv 2 Recursio o DB* P misup = 2 Which is actually FullDB* AP DB* A Output: FAP X = {F} F F F C F F C A M P F C B P F A P F C A F C F A DB* C DB* F F F DB DB* X = {F, C, A} Output: FP, CP, AP

33 I will give you back the FP-tree A FP-tree tree of DB cosists of: A fixed order amog items i DB A prefix, threaded tree of sorted trasactios i DB Header table: (item, freq, ptr) Whe used i the algorithm, the iput DB is always prued (c.f., PrueDB()) Remove ifequet items Remove ifrequet items i every trasactio

34 FP-tree Example misup = 3 TID Items bought (ordered) frequet items 100 {f, a, c, d, g, i, m, p} {f, c, a, m, p} 200 {a, b, c, f, l, m, o} {f, c, a, b, m} 300 {b, f, h, j, o, w} {f, b} 400 {b, c, k, s, p} {c, b, p} 500 {a, f, c, e, l, p, m, } {f, c, a, m, p}

35 TID Items bought (ordered) frequet items 100 {f, a, c, d, g, i, m, p} {f, c, a, m, p} 200 {a, b, c, f, l, m, o} {f, c, a, b, m} 300 {b, f, h, j, o, w} {f, b} 400 {b, c, k, s, p} {c, b, p} 500 {a, f, c, e, l, p, m, } {f, c, a, m, p} { } { } { } f : 1 f : 2 Item freq head f : 4 c : 1 c : 1 c : 2 f 4 c 4 a 3 c : 3 b : 1 b : 1 b 3 a : 1 a : 2 m 3 p 3 a : 3 p : 1 m : 1 p : 1 Isert t 1 m : 1 b : 1 p : 1 m : 1 Isert t 2 m : 2 b : 1 p : 2 m : 1 Isert all t i Output f c a b m p

36 TID frequet items 100 {f, c, a, m, p} 200 {f, c, a, b, m} 300 {f, b} 400 {c, b, p} 500 {f, c, a, m, p} p's coditioal patter base f c a m : 2 c b : Output pc { } Item freq head f 4 c 4 a 3 b 3 m 3 p 3 f : 4 c : 3 a : 3 b : 1 c : 1 b : 1 p : 1 Cleaed p s coditioal patter base C :2 C :1 m : 2 p : 2 b : 1 m : 1 STOP Header Table { } c : 3

37 TID frequet items 100 {f, c, a, m, p} 200 {f, c, a, b, m} 300 {f, b} 400 {c, b, p} 500 {f, c, a, m, p} m's coditioal patter base f c a : 2 f c a b : Output mf mc ma { } Item freq head f : 4 c : 1 f 4 c 4 a 3 c : 3 b : 1 b : 1 b 3 m 3 a : 3 { } m : 2 b : 1 ge_powerset Header Table f : 3 m : 1 Output mac maf mcf macf c : 3 a : 3

38 b's coditioal patter base f c a : 1 f : 1 c : { } Item freq head f : 4 c : 1 f 4 c 4 a 3 c : 3 b : 1 b : 1 STOP b 3 a : 3 b : 1

39 a's coditioal patter base f c : Output af ac { } Item freq head f 4 c 4 a 3 f : 4 c : 3 c : 1 a : 3 ge_powerset Output acf Header Table { } f : 3 c : 3

40 c's coditioal patter base f : 3 3 Output cf { } Item freq head f 4 c 4 f : 4 c : 3 c : 1 STOP Header Table { } f : 3

41 STOP { } Item freq head f 4 f : 4

42 FP-Growth vs. Apriori: Scalability With the Support Threshold Data set T25I20D10K D1 FP-grow th rutime D1 Apriori rutime 70 Ru time(sec.) Support threshold(%) COMP9318: Data Warehousig ad Data Miig 42

43 Why Is FP-Growth the Wier? Divide-ad-coquer: decompose both the miig task ad DB accordig to the frequet patters obtaied so far leads to focused search of smaller databases Other factors o cadidate geeratio, o cadidate test compressed database: FP-tree structure o repeated sca of etire database basic ops coutig local freq items ad buildig sub FP-tree, o patter search ad matchig COMP9318: Data Warehousig ad Data Miig 43