Regrese a predikce pomocí fuzzy asociačních pravidel

Transcription

1 Regrese a predikce pomocí fuzzy asociačních pravidel Pavel Rusnok Institute for Research and Applications of Fuzzy Modeling University of Ostrava Ostrava, Czech Republic pavel.rusnok@osu.cz March 1, 2018, Praha Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 1 / 58

2 Regression based on rules Data X 1 X n Y? FAA IF THEN RULES IF X 1 is Small THEN Y is Big IF X 2 is Small THEN Y is Very Big IF X 1 is Medium AND X 2 is Medium THEN Y is Small. Inference Mechanism Y Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 2 / 58

3 Short History Overview Fuzzy Associations 1966 Hájek et al - The GUHA method of automatic hypotheses determination 1978 Hájek and Havránek - Mechanizing Hypothesis Formation 1993 Agrawal et al - Mining association rules between sets of items in large databases Ralbovský - Fuzzy Guha Linguistic Summaries 1982 Yager - A new approach to the summarization of data 2000 Kacprzyk et al - A fuzzy logic based approach to linguistic summaries of databases Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 3 / 58

4 Association analysis Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 4 / 58

5 Association analysis example Market Baskets 1. {beer, bread, soda} 2. {beer, bread, soda, cola, diapers} 3. {beer} 4. {beer, butter, soda} 5. {beer, butter, milk} 6. {beer, bread, soda, diapers} 7. {beer, bread, soda, diapers} 8. {bread, soda, diapers} Support and Confidence supp(diapers beer)=p({diapers,beer})=3/8=0.375 conf(diapers beer)=p({diapers,beer} {diapers})=3/4=0.75 conf(beer diapers)=p({diapers,beer} {beer})=3/7=0.43 Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 5 / 58

6 Association Analysis - Four-fold table beer diapers o o o o o o o o beer beer diapers a = 3 b = 1 diapers c = 4 d = 0 a = diapers beer b = diapers beer c = diapers beer d = diapers beer Support and Confidence supp(diapers beer)=p({diapers,beer})=a/(a+b+c+d)=0.375 conf(diapers beer)=p({diapers,beer} Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních {diapers})=a/(a+b)=0.75 pravidel 6 / 58

7 Fuzzy Attributes ϕ ψ o o o o o o o o ϕ = Temperature is high 1 high Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 7 / 58

8 Semantics of Sm, Me, Bi 1 X is Sm X is Me X is Bi v L v S v R Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 8 / 58

9 Linguistic hedges Hedge Abbreviation extremely Ex significantly Si very Ve Table: Linguistic hedges with narrowing effect Hedge Abbreviation rather Ra more or less ML roughly Ro quite roughly QR very roughly VR Table: Linguistic hedges with widening effect Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 9 / 58

10 Linguistic hedges Ex Si Ve empty hedge ML Ro QR VR, }{{}}{{} narrowing effect widening effect ν functions ν a,b,c (x) = 1, c x 1 (c x)2 b x < c, (c b)(c a) (x a) 2, (b a)(c a) a x < b 0, x < a where a < b < c [0, 1] and ν a,b,c : [0, 1] [0, 1]. Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 10 / 58

11 Fuzzy sets - all hedges Figure: Ex Si Ve empty hedge ML Ro QR VR, }{{}}{{} narrowing effect widening effect 1 a Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 11 / 58

12 Fuzzy Sets Definition Fuzzy set A is a mapping from R to [0, 1]. 1 very small small medium very big a b Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 12 / 58

13 Fuzzy Attributes X o o o o o o n 0.7 ve sm X sm X me X bi X Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 13 / 58

14 Combining Fuzzy Attributes T-norm A t-norm is a function : [0, 1] [0, 1] [0, 1] which satisfies the following properties: Negation a b = b a a b a b if a a and b b (a (b c)) = ((a b) c) a 1 = a : [0, 1] [0, 1], a = 1 a T-conorm a 0 = a Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 14 / 58

15 Examples Łukasiewicz a L b = max{0, a + b 1} Product a P b = a b Minimum a M b = min{a, b} = a b Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 15 / 58

16 Residuated implication Definition A function : [0, 1] 2 [0, 1] is called an residuated implication if there exists a t-norm such that x y = sup{t [0, 1] x t y}. Łukasiewicz a L b = 1 a + b Product a P b = b/a Minimum a M b = b Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 16 / 58

17 Fuzzy Four-fold table For any two fuzzy attributes ϕ, ψ a generalized fuzzy four-fold table E(ϕ, ψ, a, b, c, d, ) can be constructed, i.e. E := ψ ψ ϕ a b ϕ c d, (1) where a = o i D ϕ(o i) a ψ(o i ), b = o i D ϕ(o i) b ψ(o i ), c = o i D ϕ(o i) c ψ(o i ), d = o i D ϕ(o i) d ψ(o i ). Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 17 / 58

18 Mining of IF THEN rules A B o o o o o o o o supp(ϕ ψ) = conf(ϕ ψ) = o i ϕ(o i ) ψ(o i ) n o i ϕ(o i ) ψ(o i ) o i ϕ(o i )? = a a+b+c+d? = a a+b Knowledge base - LD R 1,..., R n such that supp(r i ) > 0.02, and conf(r i ) > 0.9 Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 18 / 58

19 Final Inference Data X 1 X n Y? FAA IF THEN RULES IF X 1 is Small THEN Y is Big IF X 2 is Small THEN Y is Very Big IF X 1 is Medium AND X 2 is Medium THEN Y is Small. Inference Mechanism Y Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 19 / 58

20 Defuzzification Output fuzzy set 1 a defuzzified value b Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 20 / 58

21 PbLD inference input x 0 KNOWDLEDGE BASE LD R 1, R 2,..., R n {R A(x 0 ) > 0 for R := (A B)} C(y) = L i=1 (A(x 0) L B i (y)) L n {R R, s. t. A x0 A} Y Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 21 / 58

22 Implicative Fuzzy Inference - Perception based Logic Deduction Interpretation of the knowledge base ˆR(x, y) = Ordering of antecedents partial ordering N (A i (x) L B i (y)) i=1 L = {A i A j (A j x0 A i ) then (A j = A i )} Output (with x o as input) C(y) = L i=1 (A(x o) L B i (y)) L N Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 22 / 58

23 Regression based on rules Data X 1 X n Y? FAA IF THEN RULES IF X 1 is Small THEN Y is Big IF X 2 is Small THEN Y is Very Big IF X 1 is Medium AND X 2 is Medium THEN Y is Small. Inference Mechanism Y Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 23 / 58

24 Time Series Gap of economical output Intereset rate Inflation Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 24 / 58

25 Model Construction X Derived Variables X t X t 1 X t X t 1 1 (X 3 t + X t 1 + X t 2 ) Fuzzy Sets Fuzzy Association Analysis IF THEN RULES Linguistic Model Implicative Fuzzy Inference X t+1 Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 25 / 58

26 Derived Variables IR o o o o o o n 0.7 IR diff IR[i 1] IR diff [i 1] TIME n Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 26 / 58

27 Derived Variables X S(h) i (t) := X i (t h) X D(0,l,h) i (t) := X i (t h) X i (t h l) X D(1,l,h) i (t) := X D(0,l,h) i (t h) X D(0,l,h) i (t h l) X MA(n) i (t) := 1 n n X i (t j) j=1 X time (t) := t Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 27 / 58

28 Membership Degrees to Fuzzy Sets IR o o o o o o n 0.7 ve sm IR sm IR me IR bi IR Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 28 / 58

29 IF THEN Rules Example of mined rule Rule1 : IF Time is Bi AND G S(10) is Bi THEN G is Sm Rule2 : IF IR S(4) is Sm AND G S(10) is Bi THEN G is Sm Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 29 / 58

30 Comparison with DSGE model Linguistic Model DSGE Model G IR I DSGE model Linguistic model Actual values Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 30 / 58

31 Ensemble learning - models Y = f 1 (X 1,..., X m ) + E 1, Y = f 2 (X 1,..., X m ) + E 2,. Y = f r (X 1,..., X m ) + E r, W i = 1 Y f i (X 1,..., X m ), (2) Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 31 / 58

32 Ensemble learning - data X 1 X 2... X m Y F 1 F 2 o 1 e 11 e e m1 y 1 f 11 f 21 o 2 e 12 e e m2 y 2 f 12 f o N e 1N e 2N... e mn y N f 1N f 2N... F r W 1 W 2 W r f r1 w 11 w 21 w r1 f r2 w 12 w 22 w r f rn w 1N w 2N w rn Table: A table representing an extended dataset D with regression models with their weights. Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 32 / 58

33 Ensemble learning - rules Rules IF (X i is A i AND F j is A j...) THEN W i is B, for A i C X i F j C F j, B C W i (3) Ensemble Definition ens i = j w ji f ji j w ji ensmax i = f ji, where j = arg max j {1,...,r} {w ji } Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 33 / 58

34 Table: Average ranking and root mean square errors for multiple regressions for datasets with average correlation between regression models above 0.5. Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 34 / 58 Ensemble learning - Experiments Example of an rule from ensemble IF X 1 is Small AND MARS is Medium THEN W 1 is Significantly Small, LR SVM MARS KNN RT average ranking average RMSE Mean ensmax ens average ranking average RMSE

35 Time Series Prediction Ensemble IF Strength of Seasonality is Extremely Big AND Kurtosis is Quite Roughly Small THEN Weight of the ARIMA method is Big. Table: Number of Rules Generated by the Fuzzy GUHA Method and Number of Rules After Post-Processing. Method Number of Rules After Application of Algorithms Fuzzy GUHA Redundancy Removal Size Reduction ARIMA DT ES RW RWd Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 35 / 58

36 Time Series Ensemble Results Table: Average and Standard Deviation of the SMAPE Forecasting Errors. Method Error Average Error Std.Dev. ARIMA DT ES RW RWd AM FRBE Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 36 / 58

37 Too many rules Data X 1 X n Y? FAA GLOBAL MEASURES. TOO MANY RULES. REASONABLE AMOUNT OF RULES PbLD Y Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 37 / 58

38 Coverage of Data Definition cov O (LD) = 1 n n k A i (o j ) j=1 i=1 Proposition cov O (LD) [0, 1] cov O ( ) = 0 LD LD then cov O (LD ) cov O (LD) If S, T LD such that S = A C, T = B C and A B, then cov O (LD \ {S}) = cov O (LD) Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 38 / 58

39 Probabilistic coverage of data Definition cov P (LD) = f O (X 1,..., X m ) dx 1... dx m. S Estimation based on data cov P (LD) = 1 n #{o j A i (o j ) > 0} Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 39 / 58

40 Relationships Proposition Lemma cov O (LD) [0, 1] cov O ( ) = 0 LD LD then cov O (LD ) cov O (LD) If S, T LD such that S = A C, T = B C and A B, then cov O (LD \ {S}) = cov O (LD) cov O (LD) = 1 implies cov P (LD) = 1 cov P (LD) = 1 implies cov O (LD) > O Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 40 / 58

41 Algorithm for probabilistic reduction Informal description sample n rows from (training) data run n times the PbLD inference (only conditional firing) calculate the probability of a rule to be fired discard rule one by one till the desired threshold is reached descending and ascending strategies Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 41 / 58

42 Experimental Comparison FAA -> KNOWDLEDGE BASE LD supp(r i ) > 0.2, conf(r i ) > 0.9 cov O (LD ) = {1, 0.99, 0.95, 0.9} cov P (LD ) = {1, 0.99, 0.95, 0.9} Y O PbLD Y Y P Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 42 / 58

43 Experimental Results Table: Results of reduction of rules based on cov O and cov P Dataset auto-mpg automobile housing airfoil yacht FAA cov O cov P best best number error number error number error number error number error Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 43 / 58

44 Regression based on rules Data X 1 X n Y? FAA IF THEN RULES IF X 1 is Small THEN Y is Big IF X 2 is Small THEN Y is Very Big IF X 1 is Medium AND X 2 is Medium THEN Y is Small. Inference Mechanism Y Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 44 / 58

45 Fuzzy Association Analysis - experiments K-means clusters 1 1st cluster 2nd cluster 3rd cluster 0.5 min x 1 x 12 x 2 x 23 x 3 max Figure: Triangular partition derived from k-means clustering. Where x i is the mean of i-th cluster and x ij is the mean of the i-th cluster maximum and the j-th cluster minimum. Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 45 / 58

46 Fuzzy Association Analysis - examples Rules (A 1 B 1 ) and (A 2 B 2 ) conf l (A 1 B 1 ) < conf l (A 2 B 2 ) conf m (A 1 B 1 ) > conf m (A 2 B 2 ) Rules (A 3 B 3 ) and (A 4 B 4 ) conf l (A 3 B 3 ) > conf l (A 4 B 4 ) conf m (A 3 B 3 ) < conf m (A 4 B 4 ) A 1 B 1 A 2 B 2 A 3 B 3 A 4 B 4 o o Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 46 / 58

47 Fuzzy Association Analysis - experiments First n rules The first n rules mined: R n c, R n - {p, l, m} Distances K n (R n 1, R n 2 ) = 1 n (n + 1) D n (R n 1, R n 2 ) = #{R n 1 \R n 2 } r R n 1 R n 2 rank 1 (r) rank 2 (r), Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 47 / 58

48 Fuzzy Association Analysis - experimental results Table: Distances of rules with 1 antecedent mined from dataset Yeast with equi-width partition. D 200 \K 200 R 200 c R 200 l R 200 m R 200 p R 200 R 200 R 200 R 200 c x l 22 x m x 0.06 p x Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 48 / 58

49 Fuzzy Quantifier Fuzzy Quantifier q A fuzzy quantifier q is a map defined on the set of fuzzy four-fold tables, i.e. q : (R + ) 4 [0, 1], (a, b, c, d) [0, 1] From tables to classes χ χ ζ a b, ζ c d ψ ψ ϕ a b ϕ c d, Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 49 / 58

50 Classes of Fuzzy Quantifiers q is implicational a a, b b implies q(a, b ) q(a, b). q is double implicational a a, b b and c c implies q(a, b, c ) q(a, b, c). q is equivalence a a, b b, c c and d d implies q(a, b, c, d ) q(a, b, c, d). q is ratio-implicational a b ab implies q(a, b ) q(a, b). Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 50 / 58

51 Classes of Fuzzy Quantifiers - examples q is implicational q(a, b) = a a+b q is double implicational q(a, b, c) = a a+b+c q is equivalence q(a, b, c, d) = a+d a+b+c+d q is ratio-implicational q(a, b) = a, θ > 0 a+θ b Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 51 / 58

52 From fuzzy implications to fuzzy implicational quantifiers Definition A fuzzy implication is a binary operation I : [0, 1] 2 [0, 1] for which I(0, 0) = I(1, 1) = 1, I(1, 0) = 0 and x x, y y implies I(x, y ) I(x, y). Lemma Let I be fuzzy implication and ϕ 1, ϕ 2 : R [0, 1] be functions such that, for i = 1, 2, 1 ϕ i is nonincreasing, 2 ϕ i (0) = 1, and 3 ϕ i ( ) = 0. Then q I : R 2 [0, 1] defined by q I (a, b) := I(ϕ 1 (a), ϕ 2 (b)) is a fuzzy implicational quantifier. Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 52 / 58

53 Ratio-implicational quantifiers Lemma Let I be a fuzzy implication then q I (a, b) = I ( b, a a+b a+b) is a ratio-implicational quantifier with 0 = q(0, b) for all b > 0 and 1 = q(a, 0) for all a > 0. Lemma Let q be a ratio-implicational quantifier with 0 = q(0, b) for all b > 0 and 1 = q(a, 0) for all a > 0. Then there exists a fuzzy implication I q such that ( ) b q(a, b) = I q a + b, a. a + b Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 53 / 58

54 Quality measures from fuzzy four fold table When? supp(ϕ ψ) = conf(ϕ ψ) = o i ϕ(o i ) ψ(o i ) n? = o i ϕ(o i ) ψ(o i ) o i ϕ(o i ) a a + b + c + d? = a a + b ψ ψ ϕ a b ϕ c d a = o i D ϕ(o i) a ψ(o i ) b = o i D ϕ(o i) b ψ(o i ), c = o i D ϕ(o i) c ψ(o i ), d = o i D ϕ(o i) d ψ(o i ). Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 54 / 58

55 Involutively dual t-norms Definition Let 1 and 2 are two t-norms. Then we say they are involutively dual if the following holds: (x 1 y) + x 2 (1 y) = x, for x, y [0, 1]. Example ( ) Tp F (x, y) = log p 1 + (px 1)(p y 1), where p [0, ]. p 1 T F p (x, y) + T F 1/p(x, 1 y) = x. Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 55 / 58

56 Equality for involutively dual t-norms Confidence If is a and b is involutively dual t-norm then: o conf(ϕ ψ) = i ϕ(o i ) ψ(o i ) = a o i ϕ(o i ) a + b Support supp(ϕ ψ) = o i ϕ(o i ) ψ(o i ) n? = a a + b + c + d Only Product If a = b, and a + b + c + d = n then a = b = c = d = Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 56 / 58

57 Rule base measures based on single rule validity Current work Inacc = 1 O+ Σ O Σ O, Impr 1 = 1 O+ Σ, (4) O where O + and O are fuzzy subsets of O such that: O + (o) is a degree in which a rule base R is valid for o, O (o) is a degree in which a rule base R is not valid for o, Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 57 / 58

58 Conclusion and Future work Future work for you lfl: Linguistic Fuzzy Logic (R package on CRAN) > install.packages( lfl ) > frbe(d,h=10) Thank you for your attention! Questions now or pavel.rusnok@osu.cz Pavel Rusnok (IRAFM) Regrese pomocí fuzzy asociačních pravidel 58 / 58