Learning Structural Changes of Gaussian Graphical Models in Controlled Experiments

Transcription

1 Learning Structura Changes of Gaussian Graphica Modes in Controed Experiments Bai Zhang and Yue Wang Bradey Department of Eectrica and Computer Engineering Virginia Poytechnic Institute and State University Arington, VA Abstract Graphica modes are widey used in scientific and engineering research to represent conditiona independence structures between random variabes. In many controed experiments, environmenta changes or externa stimui can often ater the conditiona dependence between the random variabes, and potentiay produce significant structura changes in the corresponding graphica modes. Therefore, it is of great importance to be abe to detect such structura changes from data, so as to gain nove insights into where and how the structura changes tae pace and hep the system adapt to the new environment. Here we report an effective earning strategy to extract structura changes in Gaussian graphica mode using 1 -reguarization based convex optimization. We discuss the properties of the probem formuation and introduce an efficient impementation by the boc coordinate descent agorithm. We demonstrate the principe of the approach on a numerica simuation experiment, and we then appy the agorithm to the modeing of gene reguatory networs under different conditions and obtain promising yet bioogicay pausibe resuts. 1 Introduction Controed experiments are very common yet effective toos in scientific research. For exampe, in the studies of disease or drug effectiveness using case-contro experiments, the changes of the conditiona dependence between measurement variabes are often refected in the structura changes in the corresponding graphica modes that can revea crucia information about how the systems responds to externa stimui or adapts to changed conditions. The abiity to detect and extract the structura changes from data can faciitate the generation of new insights and new hypotheses for further studies. Consider the exampe of gene reguatory networs in systems bioogy. Gene reguatory networs are context-specific and dynamic in nature, that is, under different conditions, different reguatory components and mechanisms are activated and accordingy the topoogy of the underying gene reguatory networ changes (Zhang et a., 2009). For exampe, in response to diverse conditions in the yeast, transcription factors ater their interactions and rewire the signaing networs (Luscombe et a., 2004). Such changes in networ structures provide great insights into the underying bioogy of how the organism responses to outside stimui. In disease studies, it is important to examine the topoogica changes in transcriptiona networs between disease and norma conditions where a deviation from norma reguatory networ topoogy may revea the mechanism of pathogenesis, and the genes that undergo the most networ topoogica changes may serve as potentia biomarers or drug targets. Simiar phenomena aso appear in other areas. For instance, in web search or coaborative fitering, usefu information can be acquired by observing how certain events (e.g., aunch of an advertisement campaign) trigger changes in dependence patterns of search eywords or preference for products refected in the associated structura changes. The conditiona dependence of a set of random variabes are often mathematicay characterized by graphica modes, such as Bayesian networs and Marov networs, and various methods have been proposed to earn graphica mode structures from data (Lauritzen, 1996; Jordan, 1998). Athough earning the graphica modes under two conditions can be separatey achieved and the structura and parametric differences can be subsequenty compared, such technicay convenient framewor woud competey coapse when the

2 structura and parametric inconsistencies due to imited data sampes and noise effects are significant and hinder an accurate detection of true and meaningfu structura and parametric changes. Here we report an effective earning strategy to extract structura changes of Gaussian graphica modes in controed experiments using convex optimization. We discuss the properties of the probem formuation and introduce an efficient boc coordinate descent agorithm. We demonstrate the principe of the approach on a numerica simuation experiment, and we then appy the agorithm to the modeing of gene reguatory networs under different conditions and obtain promising yet bioogicay pausibe resuts. 2 A Revisit on Gaussian Graphica Mode Structura Learning Using 1 -reguarization The structures of graphica modes in many cases are unnown and need to be earned from data. In this paper, we focus on Gaussian graphica modes, in which the nodes (variabes) are Gaussian, and their dependence reationships are inear. Assume we have a set of p random variabes of interest, X = X 1, X 2,..., X p }, and N observations, x j = [x 1j, x 2j,..., x Nj ] T, j = 1, 2,..., p. Let X = [x 1, x 2,..., x p ] be the data matrix. Learning the structures of graphica modes efficienty is often very chaenging. Recenty, 1 -reguarization has drawn great interest in statistics and machine earning community (Tibshirani, 1996; Efron et a., 2004; Zou and Hastie, 2005; Zhao and Yu, 2006). Penaty on the 1 -norm of the regression coefficients has two very usefu properties: sparsity and convexity. The 1 -norm penaty tends to mae some coefficients exacty zeros, eading to a parsimonious soution, which naturay performs variabe seection or sparse inear mode estimation. Further, the convex nature of 1 -norm penaty maes the probem computationay tractabe, which can be soved readiy by many existing convex optimization methods. Severa 1 -reguarization approaches have been successfuy appied to graphica mode structure earning (Lee et a., 2006; Wainwright et a., 2006; Schmidt et a., 2007), especiay Gaussian graphica modes (Meinshausen and Bühmann, 2006; Banerjee et a., 2008; Friedman et a., 2008). Meinshausen and Bühmann (2006) proposed to use asso to identify the neighborhood of the nodes in Gaussian graphs. The neighborhood seection of a node X j, j = 1, 2,..., p, is soved by appying asso to earn the prediction mode of variabe X j, given a remaining variabes X j. The asso estimate ˆβ is given by ˆβ = arg min β:β j=0 x j Xβ λ β 1, (1) where λ > 0 is the Lagrange mutipier. If the th eement of ˆβ is non-zero, then there is an edge between node j and node. This procedure is performed on each of the p random variabes, and thereby the structure of the Gaussian graphica mode is earned. Meinshausen and Bühmann (2006) aso showed that under certain conditions, the proposed neighborhood seection scheme is consistent for sparse high-dimensiona graphs. 3 Probem Formuation Now we consider the probem of earning structura changes of a graphica mode between two conditions. This is equivaent to investigating how the conditiona dependence and independence of a set of random variabes change under these two conditions. Simiary, we have a set of p random variabes of interest, X = X 1, X 2,..., X p }, and we observed N 1 sampes under condition 1 and N 2 sampes under condition 2. Without oss of generaity, we assume N 1 = N 2 = N, which means we have baanced observations from two conditions. Under the first condition, for variabe X j, we have observations x (1) j = [x (1) 1j, x(1) 2j,..., x(1) Nj ]T, j = 1, 2,..., p, whie under the second condition, we have x (2) j = [x (2) 1j, x(2) 2j,..., x(2) Nj ]T, j = 1, 2,..., p. Further, et X (1) = [x (1) 1, x(1) 2,..., x(1) p ] be the data matrix under condition 1 and X (2) = [x (2) 1, x(2) 2,..., x(2) p ] be the data matrix under condition 2. Further, denote and [ ] β = = [ 1, β(1) 2 X = y j = [ x (1) j x (2) j ] [ X (1) 0 0 X (2),..., β(1) p, 1, β(2) 2, (2) ], (3),..., β(2) p ] T. (4) By ocation and scae transformations, we can aways assume that the variabes have mean 0 and unit ength, N i=1 N i=1 x (1) ij = 0, x (2) ij = 0, N i=1 N i=1 (x (1) ij )2 = 1, (x (2) ij )2 = 1, (5)

3 where j = 1, 2,..., p. We formuate the probem of earning structura changes between two conditions as a convex optimization probem. We sove the foowing optimization probem for each node (variabe) X j, j = 1, 2,..., p. f(β) = 1 2 y j Xβ 2 2 +λ 1 β 1 +λ 2 1 (6) ˆβ = arg min f(β) β 1 = arg min, 2 y j Xβ λ 1 β 1 + λ 2 1 s.t. j = 0, j = 0 (7) In (7), we earn the structures of the graphica mode under two conditions jointy. The 2 -oss function and the first 1 -reguarization term, λ 1 β 1, ead to the identification of sparse graph structure. The second 1 -reguarization term, λ 2 1, encourages sparse changes in the mode structure and parameters between two conditions, and thereby suppresses the structura and parametric inconsistencies due to noise in the data and imited sampes. The objective function (6) is non-differentiabe, continuous, and convex. The optimization probem (7) may appear simiar to the fused asso (Tibshirani et a., 2005), which was appied to protein mass spectroscopy and DNA copy number detection (Friedman et a., 2007). The fused asso encourages the fatness of the coefficient profie β j as a function of the index j. Koar et a. (2009) investigated earning of varying-coefficient varying-structure modes from time-course data, in which β t is a function of time t, and proposed a two-stage procedure that first identifies jump points and then identifies reevant covariates. The tota variation norm (TV-norm) of β t is used to encourage sparse changes aong the time course. Besides targeted at different appications, the objective function (6) has two important technica differences from the above two approaches. First, the penaty term λ 1 β 1 + λ 2 1 has boc-wise separabiity, which means the non-differentiabe objective function f(β) can be written in the form f(β) = g(β) + M h m (b m ), (8) m=1 where g(β) is convex and differentiabe, b m is some subset of β, h m (b m ) is convex and non-differentiabe, and b m1 and b m2, m 1 m 2, do not have overapping members (Tseng, 2001). We rewrite the objective function (6) as f(β) = 1 2 y j Xβ λ 1 p + λ 2 p =1 =1 ( ) ( + β(2) ) Therefore, the non-differentiabe part of f(β) can be written as the sum of p terms with non-overapping members, (, β(2) ), = 1, 2,..., p. Each (β(1), β(2) ), = 1, 2,..., p, is a coordinate boc. We wi show in Section 4 that this property is essentia for the convergence of the boc coordinate descent agorithm to sove probem (7). On the other hand, Friedman et a. (2007) has shown that coordinate-wise descent does not wor in fused asso, since the nondifferentiabe penaty function is not separabe. Additionay, the th coumn of matrix X, x, and the ( + p) th coumn of X, x +p, are orthogona, i.e., x T x +p = 0, = 1, 2,..., p. This simpifies the derivation of cosed-form soutions to the sub-probems in each iterations of the boc coordinate descent. We summarize our discussions above as three properties of probem (7). Property 1 (Convexity). The objective function (6) is continuous and convex. Property 2 (Boc-wise Separabiity). The nondifferentia part of the objective function (6), λ 1 β 1 + λ 2 1, is boc-wise separabe. Property 3 (Orthogonaity in the Coordinate Boc). x T x +p = 0, = 1, 2,..., p. To represent the resut as a graph, the non-zero eements of indicate the neighbors and edges of node X j under the first condition and the non-zero eements of indicate the neighbors and edges of node X j under the second condition.the non-zero eements of provide the changed edges (both structura and parametric difference) of node X j between two conditions. We repeat this procedure to each node X j, j = 1, 2,..., p, and then we obtain the graph under two conditions. In gene reguatory networ modeing, we are particuary interested in where and how the gene reguatory networ exhibits different networ topoogy between two conditions. To highight such changes, we extract the sub-networ in which nodes have different connections between two conditions. 4 Agorithm In the ream of computationa bioogy and data mining, vast amount of data and high dimensionaity re-

4 quire efficient agorithms. Athough the optimization probems with 1 -reguarization can be soved readiy by existing convex optimization techniques, a ot of efforts have been made to sove the probems efficienty by expoiting the specia structures of the probems. A we-nown approach is the east ange regression (LARS), which can be modified to sove asso probems (Efron et a., 2004). Recenty, coordinate-wise descent agorithms have been studied in asso reated probems, such as asso, garotte and eastic net (Friedman et a., 2007). Friedman et a. (2008) showed with experiments that a coordinate descent procedure for asso, graphica asso, is times faster than competing methods, maing it a computationay attractive method. 4.1 Boc coordinate descent agorithm In this paper, we adopt this idea and propose a boc coordinate descent agorithm to sove the optimization probem (7) for each node X j, j = 1, 2,..., p. The essence of the boc coordinate descent agorithm is one-boc-at-a-time. At iteration r + 1, ony one coordinate boc, (, β(2) ), is updated, with the remaining (, ),, fixed at their vaues at iteration r. Given β r = [,r 1,,r 2,..., β p (1),r,,r 1,,r 2,...,,r p ] T, (9) at iteration r + 1, the estimation is updated according to the foowing sub-probem β r+1 = arg min f(β) β s.t. =,r, =,r, for = 1, 2,..., p,. (10) We use a cycic rue to update parameter estimation iterativey, i.e., update parameter pair (, β(2) ) at iteration r + 1, and = ((r + 1) mod p) + 1. function of (10) as f(β) = 1 2 y j + λ 1 j, j, x,r (,r j, x (p+),r x x p ,r ) + λ 2 (,r,r ) j, + λ 1 ( + β(2) ) + λ 2( ) (11) Let ỹ j = y j j, x,r Therefore, updating ( Denote (,r+1,,r+1 ) = arg min f(β),β(2) = arg min,β(2) j, x (p+),r. (12), β(2) ) is equivaent to 1 2 ỹ j x x p λ 1 ( + β(2) ) + λ 2( ) (13) ρ 1 = ỹ j T x, (14) ρ 2 = ỹ j T x p+. (15) First, we examine a simpe case, the soution, (, β(2) ), satisfies > 0, > 0, (16) <. Tae derivative of objective function (11), and we have f f = ρ 1 + λ 1 sgn( ) + λ 2sgn( ), (17) = ρ 2 + λ 1 sgn( ) λ 2sgn( ), (18) 4.2 Cosed-form soution to the sub-probem Thus the probem is reduced to soving the subprobem (10). Since and, = 1, 2,..., p,, are fixed during iteration r+1, we rewrite the objective where sgn( ) is the sign function. When ρ 1 > λ 1 λ 2 and ρ 2 > ρ 1 + 2λ 2, we have = ρ 1 λ 1 + λ 2, = ρ 2 λ 1 λ 2. (19)

5 Simiary, we derive a cosed-form soutions to probem (10), depending on the vaues of ρ 1, ρ 2 with respect to λ 1, λ 2. The pane (ρ 1, ρ 2 ) is divided into 13 regions, as shown in Figure 1. If (ρ 1, ρ 2 ) is in region (6), then = ρ 1 + λ 1 + λ 2, = ρ 2 + λ 1 λ 2. If (ρ 1, ρ 2 ) in region (7), then (26) ρ 2 (4) ρ 2 =λ 1 + λ 2 (5) ρ 2 = (λ 1 λ 2 ) (6) ρ 2 =ρ 1 +2λ 2 (3) (0) (7) (8) (9) (10) ρ 2 =ρ 1 2λ 2 ρ 1 = (λ 1 + λ 2 ) ρ2= 2λ 1 ρ 1 ρ 1 = (λ 1 λ 2 ) 0 ρ 1 ρ 1 =λ 1 λ 2 ρ2=2λ 1 ρ 1 ρ 1 =(λ 1 + λ 2 ) (2) ρ 2 =ρ 1 +2λ 2 (1) ρ 2 =ρ 1 2λ 2 (12) ρ 2 =λ 1 λ 2 (11) ρ 2 = (λ 1 + λ 2 ) Figure 1: Soution regions of the sub-probem. Depending on the ocation of (ρ 1, ρ 2 ) in the pane, the soutions to probem (10) are as foows. If (ρ 1, ρ 2 ) is in region (0), then If (ρ 1, ρ 2 ) is in region (1), then = = 0. (20) = = 1 2 (ρ 1 + ρ 2 ) λ 1 (21) = = 1 2 (ρ 1 + ρ 2 ) + λ 1. (27) If (ρ 1, ρ 2 ) is in region (8), then = ρ 1 + λ 1 λ 2, = ρ 2 + λ 1 + λ 2. If (ρ 1, ρ 2 ) is in region (9), then = 0, = ρ 2 + λ 1 + λ 2. If (ρ 1, ρ 2 ) is in region (10), then = ρ 1 λ 1 λ 2 = ρ 2 + λ 1 + λ 2. If (ρ 1, ρ 2 ) is in region (11), then = ρ 1 λ 1 λ 2 = 0. If (ρ 1, ρ 2 ) is in region (12), then = ρ 1 λ 1 λ 2 = ρ 2 λ 1 + λ Convergence anaysis (28) (29) (30) (31) (32) Finay, we summarize the optimization procedure to sove probem (7) in Agorithm 1. If (ρ 1, ρ 2 ) is in region (2), then = ρ 1 λ 1 + λ 2, = ρ 2 λ 1 λ 2. If (ρ 1, ρ 2 ) is in region (3), then = 0, = ρ 2 λ 1 λ 2. If (ρ 1, ρ 2 ) is in region (4), then = ρ 1 + λ 1 + λ 2, = ρ 2 λ 1 λ 2. If (ρ 1, ρ 2 ) is in region (5), then = ρ 1 + λ 1 + λ 2, = 0. (22) (23) (24) (25) Agorithm 1 Boc coordinate descent agorithm to sove probem (7) Initiaization: β 0 = [0, 0,..., 0], r = 0 whie β r is not converged do (r mod p) + 1 if j then Let,r+1 =,r,,r+1 =,r, Cacuate ỹ j according to (12). Cacuate ρ 1 and ρ 2 using (14) and (15). Update,r+1 (32). end if r r + 1 end whie and,r+1, according to (20)- The convergence of Agorithm 1 is stated in the foowing theorem.

6 Theorem 1. The soution sequence generated by Agorithm 1 is bounded and every custer point is a soution of probem (7). Proof. We have shown in Property 1 and Property 2 that the objective function (6) is continuous and convex, and the non-differentia part of the objective function is boc-wise separabe. By appying Theorem 4.1 proposed by Tseng (2001), we have that the soution sequence generated by Agorithm 1 is bounded and every custer point is a soution of probem (7). 4.4 Determining parameters λ 1 and λ 2 As we discussed previousy, the first 1 -reguarization term, λ 1 β 1, eads to the identification of sparse graph structures, and the second 1 -reguarization term, λ 2 1, suppresses the inconsistencies of the networ structures and parameters between two conditions, due to the noise in the data and imited sampes. First, we consider the case λ 2 = 0. In this case, the probem (7) is equivaent to appying asso to the data under two conditions separatey. The λ 1 contros the sparsity of the earned graph, and Agorithm 1 is reduced to a coordinate descent agorithm, in which each sub-probem is asso with two orthogona predictors. The vaue of λ 1 can be determined easiy via crossvaidation. In our experiments, we used 10-fod crossvaidation, foowing steps specified in (Hastie et a., 2008). Then we consider the second parameter λ 2. The parameter λ 2 contros the sparsity of structura and parametric changes between two conditions. From regions (1) and (7) of Figure 1, we can see that if ρ 1 ρ 2 2λ 2, then and wi be set equa (Equations (21) and (27)) as the soution of the subprobem (10). Therefore, the remaining question is when ρ 1 ρ 2 is arge enough to be considered significant, at a given significance eve α. We present here a heuristic approach to determine λ 2. Appying Fisher transform to both ρ 1 and ρ 2, we have z 1 = 1 2 n 1 + ρ 1 1 ρ 1, z 2 = 1 2 n 1 + ρ 2 1 ρ 2. (33) Since data matrices X 1 and X 2 are drawn from Gaussian distributions, we now z 1 and z 2 are approximatey normay distributed with standard deviation 1 N 3 and means 1 2 n 1+ρ1 1 ρ 1 and 1 2 n 1+ρ2 1 ρ 2, respectivey. Further, under the nu hypothesis that ρ 1 = ρ 2 (and therefore z 1 = z 2 ), define z = z 1 z 2, (34) which approximatey foows norma distribution with 1 zero mean and standard deviation. (N 3)/2 At a given significance eve α (e.g., α = 0.01 is used in Section 5), if z = z 1 z 2 s, it wi be considered significant, where s = Φ 1 (1 α/2)/ (N 3)/2. Through simpe derivation, we have z = z 1 z 2 s ρ 1 ρ 2 e2s 1 e 2s + 1 (1 ρ 1ρ 2 ) = 2λ 2 (35) To further simpify (35) with some approximation, we estimate overa ρ 1 ρ 2 by ρ 1 ρ 2 = 2 j< yt j x y T j x p+/p(p 1). Substituting ρ 1 ρ 2 in (35), we have 5 Experiments λ 2 = e2s 1 2e 2s + 2 (1 ρ 1ρ 2 ). (36) 5.1 A synthetic experiment We first use a synthetic exampe to iustrate the principe and test the proposed method. Assume there are six nodes in the Gaussian graphica mode, A, B, C, D, E, F. Under condition 1, their reationships are represented by Figure 2a. Under condition 2, their reationships are atered, as shown in Figure 2b. We generated 200 sampes from the joint Gaussian distribution according to the Gaussian graphica mode with the structure specified by Figure 2a, and 200 sampes from the joint Gaussian distribution according to Gaussian graphica mode with the structure specified by Figure 2b. The penaty parameters are set to λ 1 = 0.22 and λ 2 = 0.062, cacuated according to Section 4.4. Figure 3a is the composite networ under two conditions inferred by the proposed agorithm, where the bac ines are the edges that exist under both conditions, the red ines are the edges that exist ony under condition 1 and the green ines are the edges that exist ony under condition 2. Since we are more interested in the changed part of the graph, we extracted the edges and nodes invoved in the changes to highight these structura changes. We term it differentia sub-networ, as shown in Figure 3b. We can see the proposed agorithm accuratey captured the structura changes of the graphica mode between two conditions. 5.2 Experiment on modeing gene reguatory networs under two conditions Inference of the structures of gene reguatory networs from expression data is a fundamenta probem in com-

7 prised of nodes MBP1 SWI6, CLB5, CLB6, PHO2, FLO1, FLO10 and TRP4 and the green and red ines is the focus of our study that our agorithm tries to identify from expression data. (a) Condition 1. (b) Condition 2. Figure 2: The structures of the Gaussian graphica mode under two conditions. (a) Composite networ. (b) Differentia networ. sub- Figure 3: The networ structure earned by the proposed method. The bac ines are the edges that exist under both conditions. The red ines are the edges that exist ony under condition 1. The green ines are the edges that exist ony under condition 2. putationa bioogy. Our goa here is to infer and extract the structura changes of a gene reguatory networ between two conditions using gene expression data. SynTReN is a networ generator that creates synthetic transcriptiona reguatory networs and produces simuated gene expression data that approximate experimenta data, used as benchmars for the vaidation of bioinformatics agorithms (Van den Buce et a., 2006). To test the appicabiity of the proposed framewor in gene reguatory networ modeing, we used the software SynTReN to generate one simuation dataset of 50 sampes of a sub-networ drawn from an existing signaing networ in Saccharomyces cerevisiae. Then we changed part of networ and used SynTReN to generate another dataset of 50 sampes according to this modified networ. The networs under two conditions is shown in Figure 4a. The networ contains 20 nodes that represent 20 genes. The bac ines indicate the reguatory reationships that exist under both conditions. The red and green ines are the reguatory reationships that exist ony under condition 1 and condition 2, respectivey. The sub-networ com- Figure 4b shows the differentia sub-networ between the two conditions extracted by the proposed agorithm. The penaty parameters are set to λ1 = 0.28 and λ2 = 0.123, cacuated according to Section 4.4. Compared with the nown networ topoogy shown in Figure 4a, the proposed agorithm correcty identified a the nodes with structura changes and 7 of 10 differentia edges. The edge between CDC10 and ACE2 was fasey detected. This indicates that our agorithm can successfuy detect these interesting genes using their networ structure information, even though the means of their expressions did not change substantiay between the two conditions. Therefore, this method is abe to identify biomarers that cannot be detected by traditiona gene raning methods, providing a compimentary approach for biomarer identification probem. 6 Concusions In this paper, we reported an effective earning strategy to extract structura changes in Gaussian graphica modes in controed experiments. We presented a convex optimization framewor using `1 -reguarization to formuate this probem, and introduced an efficient boc coordinate descent agorithm to sove it. We demonstrated the effectiveness of the approach on a numerica simuation experiment, and then we appied the agorithm to detecting gene reguatory networ structura changes under two conditions and obtained very promising resuts. Acnowedgements This wor was supported in part by the Nationa Institutes of Heath under Grants NS and CA References Banerjee, O., E Ghaoui, L., and d Aspremont, A. (2008). Mode seection through sparse maximum ieihood estimation for mutivariate gaussian or binary data. Journa of Machine Learning Research, 9, Efron, B., Hastie, T., Johnstone, I., and Tibshirani, R. (2004). Least ange regression. The Annas of Statistics, 32(2), Friedman, J., Hastie, T., Ho fing, H., and Tibshirani, R. (2007). Pathwise coordinate optimization. The Annas of Appied Statistics, 1(2),

8 (a) (b) Figure 4: (a) The gene reguatory networ under two conditions. Nodes in the networ represent genes. Lines in the networ indicate reguatory reationships between genes. The bac ines are the reguatory reationships that exist under both conditions. The red and green ines represent the reguatory reationships that exist ony under condition 1 and under condition 2, respectivey. (b) The sub-networ extracted by the proposed agorithm. Friedman, J., Hastie, T., and Tibshirani, R. (2008). Sparse inverse covariance estimation with the graphica asso. Biostatistics, 9(3), Hastie, T., Tibshirani, R., and Friedman, J. (2008). The Eements of Statistica Learning: Data Mining, Inference, and Prediction, Second Edition. Springer Series in Statistics. Springer, 2nd ed. edition. Jordan, M. (1998). Learning in Graphica Modes. The MIT Press. Koar, M., Song, L., and Xing, E. P. (2009). Sparsistent earning of varying-coefficient modes with structura changes. In Advances in Neura Information Processing Systems (NIPS 2009). Lauritzen, S. L. (1996). Graphica Modes (Oxford Statistica Science Series). Oxford University Press, USA. Lee, S.-I., Ganapathi, V., and Koer, D. (2006). Efficient structure earning of Marov networs using L1-reguarization. In Advances in Neura Information Processing Systems (NIPS 2006). Luscombe, N. M., Madan Babu, M., Yu, H., Snyder, M., Teichmann, S. A., and Gerstein, M. (2004). Genomic anaysis of reguatory networ dynamics reveas arge topoogica changes. Nature, 431(7006), Meinshausen, N. and Bühmann, P. (2006). Highdimensiona graphs and variabe seection with the asso. The Annas of Statistics, 34(3), Schmidt, M., Nicuescu-Mizi, A., and Murphy, K. (2007). Learning graphica mode structure using L1-reguarization paths. In AAAI 07: Proceedings of the 22nd nationa conference on Artificia inteigence, pages AAAI Press. Tibshirani, R. (1996). Regression shrinage and seection via the asso. Journa of the Roya Statistica Society, Series B, 58, Tibshirani, R., Saunders, M., Rosset, S., Zhu, J., and Knight, K. (2005). Sparsity and smoothness via the fused asso. Journa of the Roya Statistica Society Series B, 67(1), Tseng, P. (2001). Convergence of a boc coordinate descent method for nondifferentiabe minimization. Journa of Optimization Theory and Appications, 109(3), Van den Buce, T., Van Leemput, K., Naudts, B., van Remorte, P., Ma, H., Verschoren, A., De Moor, B., and Marcha, K. (2006). SynTReN: a generator of synthetic gene expression data for design and anaysis of structure earning agorithms. BMC Bioinformatics, 7(1), 43+. Wainwright, M. J., Raviumar, P., and Lafferty, J. D. (2006). High-dimensiona graphica mode seection using L1-reguarized ogistic regression. In Advances in Neura Information Processing Systems (NIPS 2006). MIT Press. Zhang, B., Li, H., Riggins, R. B., Zhan, M., Xuan, J., Zhang, Z., Hoffman, E. P., Care, R., and Wang, Y. (2009). Differentia dependency networ anaysis to identify condition-specific topoogica changes in bioogica networs. Bioinformatics, 25(4), Zhao, P. and Yu, B. (2006). On mode seection consistency of asso. Journa of Machine Learning Research, 7, Zou, H. and Hastie, T. (2005). Reguarization and variabe seection via the eastic net. Journa of the Roya Statistica Society B, 67,