Exercises Advanced Data Mining: Solutions

Size: px

Start display at page:

Download "Exercises Advanced Data Mining: Solutions"

Shanon Moody
6 years ago
Views:

1 Exercises Advaced Data Miig: Solutios Exercise 1 Cosider the followig directed idepedece graph a) Give the factorizatio of P (X 1, X 2,..., X 9 ) correspodig to this idepedece graph. P (X) = 9 P (X i X pa(i) ) i=1 = P (X 1 )P (X 2 )P (X X 1, X 2 )P (X )P (X 5 X 2 )P (X 6 X ) P (X 7 X, X )P (X 8 X 6, X 7 )P (X 9 X 5, X 7 ) Costruct the appropriate moral graphs to check whether the followig coditioal idepedeces hold: b) 6 7 To verify X Y Z, take the directed idepedece graph o a + (X Y Z) ad moralize this graph. The you ca verify the idepedece property i the resultig udirected graph usig separatio. 1

2 The directed idepedece graph o a + ({6, 7}) is give left, the correspodig moral graph is give right: Sice 6 ad 7 are ot separated by the empty set (there is a path betwee 6 ad 7), they are ot margially idepedet. c) 6 7 For the graphs, see b). Yes, every path betwee 6 ad 7 must pass through. d) The directed idepedece graph o a + ({6, 7, 8}) is give left, the correspodig moral graph is give right: 8 8 No, 8 does ot separate 6 ad 7 i the moral graph. e) 2 9 {5, 7} The directed idepedece graph o a + ({2, 5, 7, 9}) is give left, the correspodig moral graph is give right: 2

3 Yes: {5,7} separates 2 from 9, that is, every path from 2 to 9 must pass through a ode i the set {5,7}. f) 2 9 {, 5} For the graphs, see e). Yes: {,5} separates 2 from 9. g) 5 8 The directed idepedece graph o a + ({5, 8}) is give left, the correspodig moral graph is give right: No, there is a path betwee 5 ad 8. h) 5 8 For the graphs, see g). Yes, separates 5 from 8 i the moral graph. Exercise 2 I structure learig of Bayesia etworks oe ofte uses a score fuctio to determie the quality of a etwork structure, i combiatio with a hill-climbig local search strategy. Oe popular score fuctio is BIC (Bayesia Iformatio Criterio): BIC(M) = L(M) l 2 dim(m),

4 where L(M) deotes the value of the loglikelihood fuctio of model M, evaluated at the maximum, dim(m) deotes the umber of parameters of model M, ad deotes the umber of observatios i the data set. We wat to costruct a model o the followig data set o biary variables: X 1 X 2 X The iitial model i the search is the mutual idepedece model (correspodig to the empty graph). a) Give the loglikelihood fuctio for the mutual idepedece model o this data set. L = 5 log p 1 (0)+5 log p 1 (1)+6 log p 2 (0)+ log p 2 (1)+5 log p (0)+5 log p (1), where p 1 (0) is shorthad for p(x 1 = 0). b) Give the maximum likelihood estimates for the parameters of the mutual idepedece model. ˆp 1 (0) = 1(0) ˆp 2 (0) = 2(0) ˆp (0) = (0) = 5 = 6 = 5 ˆp 1 (1) = 1(1) ˆp 2 (1) = 2(1) ˆp (1) = (1) = 5 = = 5 c) Compute the value of the loglikelihood fuctio obtaied by fillig i the maximum likelihood estimates computed i b) i the likelihood fuctio give i a). Use the atural logarithm i your computatios, ad use the covetio that 0 l 0 = 0. L = 5 log log log 6 + log + 5 log log

5 d) Give all eighbours of the curret model, assumig a eighbour ca be obtaied by either: addig a edge, removig a edge, or reversig a edge. Which of these eighbour models are equivalet? Pairs of models i the same row are equivalet, because moralisatio does ot require marryig parets, ad the resultig udirected graphs are the same. e) Would addig a edge from X 1 to X 2 (or vice versa) improve the BIC score? Explai. No, X 1 ad X 2 are idepedet i the data: ˆP (X2 ) = ˆP (X 2 X 1 ). This meas that addig a edge from X 1 to X 2 does ot improve the loglikelihood score. The BIC-score will go dow because of the extra parameter. f) Cosider the eighbour model obtaied by addig a edge from X 1 to X. Is this model preferred to the iitial model o the basis of the BIC-score? Explai. We compute ˆp 1 (0 0) = 1 5 ˆp 1 (1 0) = 5 ˆp 1 (0 1) = 5, ˆp 1(1 1) = 1 5 where ˆp 1 (0 0) is shorthad for ˆp(x = 0 x 1 = 0). The ew loglikelihood score becomes L = 5 log log log 6 + log + log log 5 + log 5 + log

6 The loglikelihood score improves by = 1.9. This is at the cost of oe extra parameter that costs 0.5 log = All i all addig a edge from X 1 to X improves the BIC score by = Exercise We are costructig a model o the followig data set o biary variables: X 1 X 2 X X Suppose the curret model i the search has the followig structure: X 1 X 2 X X a) Give the loglikelihood fuctio for the give model ad data set. L = log p 1 (0) + 6 log p 1 (1) + 5 log p 2 (0) + 5 log p 2 (1) + 0 log p 12 (0 0, 0) + 2 log p 12 (1 0, 0) + log p 12 (0 0, 1) + log p 12 (1 0, 1) + log p 12 (0 1, 0) + 0 log p 12 (1 1, 0) + log p 12 (0 1, 1) + 2 log p 12 (1 1, 1) + 2 log p (0 0) + log p (1 0) + log p (0 1) + log p (1 1) b) Give the maximum likelihood estimates for the model parameters. See aswer to c). 6

7 c) Compute the value of the loglikelihood fuctio (evaluated at the maximum) for the give model ad data set. Use the atural logarithm i your computatios, ad use the covetio that 0 l 0 = 0. L = log + 6 log log log log log 1 + log + log + log log 0 + log log log log 5 + log 5 + log 1 5 = d) Compute the BIC score of this model o the give data set ( ) = e) Give all eighbours of the curret model, assumig a eighbour ca be obtaied by either: addig a edge, removig a edge, or reversig a edge. Which of these eighbour models are equivalet? Addig a arc: A B C D A ad B are equivalet. Removig a arc: E F G Reversig a arc: H I J 7

8 H ad I are equivalet. f) Cosider the eighbour model obtaied by addig a edge from X 1 to X. Is this model preferred to the curret model? Explai. The paret set of X chages so we have to recompute the part of the score correspodig to this ode. The boxed part of the loglikelihood uder c) is replaced by 2 log log 2 + log + log.16 The boxed part uder c) evaluates to 5.86 so the loglikelihood icreases by 1.7. This is however at the cost of two extra parameters, that cost 1.15 each, so all i all additio of a arc from X 1 to X decreases the BIC score. Hece it is ot preferred to the curret model. Exercise Frequet Item Set Miig a) Level 1: cadidate support frequet? A Y B 5 Y C 2 Y D Y E 2 Y All 1-itemsets are frequet, so all 2-itemsets are cadidates at level 2: cadidate support frequet? AB Y AC 1 N AD 1 N AE 0 N BC 2 Y BD Y BE 2 Y CD 1 N CE 1 N DE 2 Y All subsets of BDE are frequet, so this is a cadidate at level : cadidate support frequet? BDE 2 Y 8

9 There are o level cadidates. b) A itemset is a geerator if it is frequet ad has o subset with the same support. Level 1: cadidate support geerator? A Y B 5 Y C 2 Y D Y E 2 Y All 1-itemsets are geerators, so all 2-itemsets are cadidates at level 2: cadidate support geerator? AB N AC 1 N AD 1 N AE 0 N BC 2 N BD N BE 2 N CD 1 N CE 1 N DE 2 N Now we have all geerators. The ext step is to compute their closure. The closure has the same support as the geerator. The result is the set of all closed frequet itemsets. geerator closure support A AB B B 5 C BC 2 D BD E BDE 2 c) If X Y ad s(x) = s(y ), the σ(x) = σ(y ). Hece c(x) = f(σ(x)) = f(σ(y )) = c(y ). d) This follows from the defiitio of the closure operator. The closure of a item set X is the set of items Y X that is cotaied i all trasactios that cotai X. So if c(x) = Y, the σ(x) = σ(y ). It follows that s(x) = s(y ). e) I d) we showed that if c(x) = Y, the σ(x) = σ(y ). Hece c(x) = f(σ(x)) = f(σ(y )) = c(y ) = Y. 9

10 Exercise 5: Classificatio Trees a) i(t 1 ) = = 0.2, i(t 2 ) = = , i(t ) = = 0.25, i = = b) T 1 is obtaied by pruig i t 7 : the resultig tree has the same resubstitutio error as T max. The we compute for each iteral ode t: We have g(t 1 ) = g(t) = R(t) R(T t) T t , g(t 2 ) = = 0.05, g(t ) = The miimum is g(t 2 ), so we prue i t 2. The we recompute the g values. g(t ) does t have to be recomputed because we did t prue below t (i.e. T t has t chaged), ad g(t 1 ) = = 0.1 which is smaller tha g(t ), so we prue i the root. Summarizig: T 1 is obtaied by pruig T max i t 7. This is the best tree for α [0, 0.05). T 2 is obtaied by pruig T 1 i t 2. This is the best tree for α [0.05, 0.1). The we prue i the root, which is the best tree for α [0.1, ). = 0.2

ECE 8527: Introduction to Machine Learning and Pattern Recognition Midterm # 1. Vaishali Amin Fall, 2015

ECE 8527: Introduction to Machine Learning and Pattern Recognition Midterm # 1. Vaishali Amin Fall, 2015 ECE 8527: Itroductio to Machie Learig ad Patter Recogitio Midterm # 1 Vaishali Ami Fall, 2015 tue39624@temple.edu Problem No. 1: Cosider a two-class discrete distributio problem: ω 1 :{[0,0], [2,0], [2,2],