Learning Gaussian Graphical Models with Unknown Group Sparsity

Transcription

1 Learning Gaussian Graphical Models with Unknown Group Sparsity Kevin Murphy Ben Marlin Depts. of Statistics & Computer Science Univ. British Columbia Canada

2 Connections Graphical models Density estimation Signal processing L1 etc L1 etc Optimization

3 Connections Hierarchical Bayes Structured sparsity Structured sparsity Graphical models Density estimation Signal processing L1 etc L1 etc Optimization

4 Structured sparsity Sparse models are often more interpretable and yield better prediction accuracy than dense ones Sometimes there is structure in the sparsity pattern itself Eg. Multi-task feature selection

5 Graphical model structure learning Ecoli compendium experiments genes 4 n=1211, d=445 Goals: Learn interpretable structure Build better density models for high dimensional data

6 Structured sparsity in GM learning Similar types of nodes within a graph may share similar neighbors (stochastic blocks model) This talk Multi-level analysis: individual graph = population graph plus random effects Work in progress

7 Gaussian graphical models (GGMs) Graph encodes structural zeros in precision matrix (conditional independence relationships) p(x G,µ,Ω 1 )=N(x µ,ω 1 ), Ω jk =0 G jk =0 Ω= Ω 11 Ω Ω 21 Ω 22 Ω Ω 32 Ω 33 Ω Ω 43 Ω 44 X1 X2 X3 X4 X j X j X Nj Markov property Dempster, Speed, Lauritzen, et al.

8 Maximum likelihood estimation To find the MLE given known G, maximize the following convex objective p(d Ω) Ω n/2 exp( 1 2 n (x i x) T Ω(x i x)) i=1 = Ω n/2 exp( n 2 tr(sω)) l(ω) def = 2 n logp(d Ω)=logdetΩ tr(sω) Subject to Ω 0,Ω jk =0 G jk =0

9 Outline Previous work: L1 and group L1 approaches to learning structure Our contribution: unknown groups

10 Sparse Cholesky Decomposition Given a known ordering, we can learn a sparse DAG, with regression weights W X j =µ j + k π j w jk (X k µ k )+ v j Z j Σ 1 = (I W) T D 1 (I W) def = T T D 1 T T = 1 w 21 1 w 32 w w d1 w d2... w d,d 1 1

11 Sparse dependency networks We can regress each node on all the others, and impose an L1 regularizer (Meinshausen & Buhlmann) ie we optimize the penalized pseudo likelihood N n=1 D p(x nj x n, j,w j,σj 2 )+λ w j,: 1 j=1

12 Graphical lasso Put L1 penalty on precision matrix max Ω 0 logdet(ω) tr(sω) D i=1 Semi-definite program. D λ ij Ω ij j=1 λ jj 0,λ max jk = ˆΣ jk Banerjee proposed a coordinate descent method, where we optimize one row/column at a time by solving a QP, O(T d 4 ) time, T=#iter. Friedman et al. improved this by showing that each subproblem is a weighted lasso problem. Still O(T d 3 ). Banerjee et al, Friedman et al, Lee et al.

13 Group graphical lasso Sometimes nodes can be grouped, eg genes in the same pathway. We would like to learn a graph which is block sparse. We can use the group L1 method to get a new convex objective: max Ω 0 logdet(ω) tr(sω) k Common p-norms: v = max v i i v 2 = i v 2 i λ k,l {Ω zk,z l } p l Duchi et al, UAI 2008

14 Outline Previous work: L1 and group L1 approaches Our contribution: unknown groups

15 Stochastic blocks model (SBM) We associate nodes with latent types or groups, z i {1,,K} The probability of an edge from node i to node j depends on their types (say red and blue), and on the probability of a red blue edge. G i,j Ber(π zi,z j ) z i Mu(θ,1) π k,k Beta(a k,k,b k,k ) θ Dir( α K ) Nowicki & Snijders

16 Using SBM to define the prior on Ω The SBM generates a block sparse undirected graph G via assignments z We would like G to induce a block sparse penalty matrix on Ω. But the normalization constant is intractable to compute (not a problem for fixed z). Graphs not just independent regressions! p(ω λ,z) = 1 Z pd(ω) exp( λ k,l Ω zk,z l p ) k l Z(λ,z) = exp( λ k,l Ω zk,z l p )dω pd k l

17 2 approaches to avoid intractable Z Use dependency networks Lower bound the objective

18 Block sparse dependency networks Dependency networks are a very fast way to learn graph structures. They do not define a valid joint density, but can be useful for data visualization. It is simple to impose block sparsity on them: use a SBM to define G, and use this indicator matrix to define a spike and slab prior G SBM w i,j N(0,σ 2 0) G i,j N(0,σ 2 1) 1 G i,j x i,n N(w T ix i,n,σ 2 )

19 Full model

20 Inference We use variational mean field, with a point estimate of W (which is O(D 2 ) in size)

21 Output of inference Clustering Graph Abstract graph Nuisance variables p(z i =k) = φ ik p(g i,j =1) = γ i,j p(π k,l ) = Beta(a k,l,b k,l) Ŵ,p(σ),p(θ)

22 Computational speed At each iteration, we solve D related ridge regression problems, each of size D, costs O(D 4 ). No obvious way to use standard matrix tricks to improve this (invert diagonal + up/downdate) So we do adaptive scheduling: only update top 10% of nodes, sorted by Λ d In Matlab: for N=174,D=667, was 50 sec per iter., now is 5 sec

23 Model selection We use spectral clustering to propose splits The graph for cluster k is defined by S = {i:z i =k} H = W(S,S) +0.5 W(S,S) W(S,S) T We propose splitting each cluster and use the variational free energy as an objective to select the best

24 Estimating Ω Once we have estimated the depnet, we want to convert it to a GGM so we have a proper joint density model We use the inferred clusterings argmax k p(z d =k) to define the groups, and use group L1 penalized MAP estimation to estimate Ω. We estimate λ by CV.

25 Results on mocap data D=60 (20 markers in 3d), N=100

26 Test set likelihood vs λ for different N Baseline ˆΣ=S+ǫI

27 Results on gene expression data N=174,D=667

28 2 approaches to avoid intractable Z Use dependency networks Lower bound the objective

29 Z (Independent L1) Between within

30 Upper bound for Z We integrate over symmetric matrices with positive elements on the diagonal.

31 EM algorithm Work in progress

32 Conclusions Lots of recent interest in sparse GGMs using convex optimization Combining clustering and sparse graph estimation results in a non-convex problem (cf clustered multitask learning) However, (variational) EM can be used, and can leverage the convex solvers in the M step Still looking for killer apps