On the Difficulty of Selecting Ising Models with Approximate Recovery

Transcription

1 On the Difficulty of Selecting Ising Moels with Approxiate Recovery Jonathan Scarlett an Volkan Cevher Abstract In this paper, we consier the proble of estiating the unerlying graphical oel of an Ising istribution given a nuber of inepenent an ientically istribute saples. We aopt an approxiate recovery criterion that allows for a nuber of isse eges or incorrectly-inclue eges, thus eparting fro the extensive literature consiering the exact recovery proble. Our ain results provie inforation-theoretic lower bouns on the require nuber of saples i.e., the saple coplexity for graph classes iposing constraints on the nuber of eges, axial egree, an sparse separation properties. We ientify a broa range of scenarios where, either up to constant factors or logarithic factors, our lower bouns atch the best known lower bouns for the exact recovery criterion, several of which are known to be tight or near-tight. Hence, in these cases, we prove that the approxiate recovery proble is not uch easier than the exact recovery proble. Our bouns are obtaine via a oification of Fano s inequality for hanling the approxiate recovery criterion, along with suitably-esigne ensebles of graphs that can broaly be classe into two categories: i Those containing graphs that contain several isolate eges or cliques an are thus ifficult to istinguish fro the epty graph; ii Those containing graphs for which certain groups of noes are highly correlate, thus aking it ifficult to eterine precisely which eges connect the. We support our theoretical results on these ensebles with nuerical experients. I. INTRODUCTION Graphical oels are a wiely-use tool for proviing copact representations of the conitional inepenence relations between rano variables, an arise in areas such as iage processing [], statistical physics [], coputational biology [3], natural language processing [4], an social network analysis [5]. The proble of graphical oel selection consists of recovering the graph structure given a nuber of inepenent saples fro the unerlying istribution. While this funaental proble is NP-har in general [6], there exist a variety of ethos guaranteeing exact recovery with high probability on restricte classes of graphs, such as boune egree an boune nuber of eges. Existing works have focuse priarily on Ising oels an Gaussian oels, an our focus in this paper is on the forer. In particular, we focus in the proble of approxiate recovery, in which one can tolerate soe nuber of isse eges or incorrectly-inclue eges. The otivation for such a stuy is that the exact recovery criterion is very restrictive, an not soething that one woul typically expect to achieve The authors are with the Laboratory for Inforation an Inference Systes LIONS, École Polytechnique Féérale e Lausanne EPFL e-ail: {jonathan.scarlett,volkan.cevher}@epfl.ch. This work was supporte in part by the European Coission uner Grant ERC Future Proof, SNF an SNF CRSII-47633, an EPFL Fellows Horizon00 grant in practice. Wiely-stuie oels such as the Ising oel are often inaccurate in practical applications, which ay further liit the utility of results on exact recovery. Our ain focus is on algorith-inepenent lower bouns for Ising oels, revealing the nuber of easureents require for approxiate recovery regarless of the coputational coplexity. The challenges that we aress inclue extening Fano s inequality [7], [8] to the case of approxiate recovery, an applying it to restricte sets of graphs that prove the ifficulty of approxiate recovery. In particular, in aressing the latter of these, we note that the best converse bouns for exact recovery [7], [8] are base on consiering graphs that iffer in only one or two eges, aking the ifficult to istinguish. This approach cannot be use for approxiate recovery, since we use a criterion in which significantly ore than one or two ege errors are allowe. Our ain results reveal a broa range of graph classes for which the approxiate recovery lower bouns exhibit the sae scalings as the best-known exact recovery lower bouns [7], [8], which are known to be tight or near-tight in any cases of interest. This suggests that, at least for the classes that we consier, the approxiate recovery proble is not uch easier than the exact recovery proble. A. Proble Stateent The ferroagnetic Ising oel [9] is specifie by a graph G = V, E with vertex set V = {,..., p} an ege set E. Each vertex is associate with a binary rano variable X i {, }, an the corresponing joint istribution is where P G x = Z exp i,j λ ij = { λ i, j E 0 otherwise, λ ij x i x j, an Z is a noralizing constant calle the partition function. Here λ > 0 is a paraeter to the istribution, soeties calle the inverse teperature. Given a atrix X {0, } n p of n inepenent saples fro this istribution, an estiator or ecoer constructs an estiate Ĝ of G, or equivalently, an estiate Ê of E. Recovery Criterion: Given soe class G of graphs, the wiely-stuie exact recovery criterion seeks to characterize P e := ax P[Ê E]. 3 G G We instea consier the following approxiate recovery criterion, for soe axiu nuber of errors q ax 0: P e q ax := ax G G P[ E Ê > q ax ], 4

2 where E Ê = E\Ê Ê\E. In this efinition, q ax oes not epen on G, an hence, the nuber of allowe ege errors oes not epen on the graph itself. We consier graph classes with a axiu nuber of eges equal to soe value k, an set q ax = θ k for soe constant θ 0, not scaling with the proble size. Note that θ = woul trivially give P e = 0. Graph Classes: We consier the following three neste classes of graphs G k,,η,γ G k, G k : Ege boune class G k This class contains all graphs with at ost k eges. Ege an egree boune class G k, This class contains the graphs in G k such that each noe has egree i.e., nuber of eges it is involve in at ost. Sparse separator class G k,,η,γ This class contains the graphs in G k, satisfying the η, γ-separation conition [0]: For any two non-connecte vertices in the graph, one can siultaneously block all paths of length γ or less by blocking at ost η noes. The restriction on the nuber of eges is otivate by the fact that real-worl graphs are often sparse. The restriction on the egree is also relevant in applications, an is particularly coonly-assue in the statistical physics literature. The sparse separation conition is soewhat ore technical, but it is of interest since it is known to perit polynoial-tie exact recovery in any cases [0], []. Generalize Ege Weights: A generalization of the above Ising oel allows λ ij to take ifferent non-zero values for each i, j E, soe of which ay be negative. Previous works consiering oel selection for this generalize oel have sought iniax bouns with respect to the graph class an these paraeters subject to λ in λ ij λ ax for soe λ in an λ ax. The lower bouns erive in this paper ieiately iply corresponing lower bouns for this generalize setting, provie that our paraeter λ lies in the range [λ in, λ ax ]. Notation an Terinology: Throughout the paper, we let P G an E G enote probabilities an expectations with respect to P G e.g., P G [X i = X j ], E[X i X j ]. We enote the floor function by, an the ceiling function by. We use the stanar terinology that the egree of a noe v V is the nuber of eges in E containing v, an that a clique is a subset C V of size at least two within which all pairs of noes have an ege between the. B. Relate Works A variety of algoriths with varying levels of coputational efficiency have been propose for selecting Ising oels with rigorous guarantees, incluing conitional inepenence tests for caniate neighborhoos [], correlation tests in the presence of sparse separators [0], [3], greey techniques [4] [7], convex optiization approaches [8], eleentary estiators [9], an intractable inforationtheoretic techniques [7]. These works have ae various assuptions on the unerlying oel, incluing incoherence assuptions [8], [9] an long-range correlation assuptions [0], [3]. A notable recent work avoiing these is [7], which provies recovery guarantees using an algorith whose coplexity is only quaratic in the nuber of noes for a fixe axiu egree, thus resolving an open question pose in [0]. Early works proviing algorith-inepenent lower bouns use only graph-theoretic properties [0], [], []; the resulting bouns are loose in general, since they o not capture the effects of the paraeters of the joint istribution e.g., λ. Several refine bouns were given in [7] for graphs with a boune egree or a boune nuber of eges. Aitional classes were consiere in [8], incluing the boune girthclass an a class relate to the separation criterion of [0] an hence relate to G k,,η,γ efine above. While our techniques buil on those of [7], [8], we ust consier significantly ifferent ensebles, since those in [7], [8] contain graphs that iffer only by one or two eges, thus aking approxiate recovery trivial. Beyon Ising oels, several works have provie necessary an sufficient conitions for recovering Gaussian graphical oels [], [] [5]. In this context, one necessary conition for approxiate recovery appeare [, Cor. 7], but the corresponing assuptions an techniques use were vastly ifferent to ours: The rano Erös-Rényi oel was consiere instea of a eterinistic class, an an aitional walk-suability conition specific to the Gaussian oel was ipose. The necessary conitions for list ecoing [6] bear soe siilarity to approxiate recovery, but the proble an its analysis are in fact uch ore siilar to exact recovery, allowing all of the ensebles fro [7], [8] to be applie irectly. C. Contributions Our ain results, an the corresponing existing results for exact recovery, are suarize in Table I, where we provie necessary scaling laws on the nuber of saples neee to obtain a vanishing probability of error P e q ax. Note that soe of the exact recovery conitions given in the final colun were not explicitly given in [7], [8], but they can easily be inferre fro the proofs therein; see Section II for further iscussion. We also observe that our analysis involves hanling ore cases separately copare to [7], [8]; in those works, the final three rows corresponing to G k in Table I are all a single case giving Ωk log p scaling, an siilarly for G k,. We ake the following observations: In all of the known cases where exact recovery is known to be ifficult, i.e., exponential in a quantity that increases in the proble iension, the sae ifficulty is observe for approxiate recovery, at least for the values of q ax shown. For G k an G k,, this is true even when we allow for up to a quarter of the eges to be in error. Note that we i not seek to optiize this fraction in our analysis, an we expect siilar ifficulties to arise even when higher proportions of errors are allowe. In fact, by a siple variation of our analysis outline in Reark in Section IV-C, we can alreay increase this fraction fro 4 to.

3 3 Graph Class Boune ege G k Distortion q ax < k 4 Theores an Boune ege an egree G k, Distortion q ax < k 4 Theores 3 an 4 Boune ege an egree with sparse separators G k,,η,γ cη Distortion q ax < k ηη+γ+ c 0,, {0,..., η} Theore 5 Necessary for approxiate recovery this paper Best known necessary for exact recovery [7], [8] Paraeters λ = ω k Exponential in λ k Exponential in λ k λ = O k Ωk log p k p k Ωk log p λ = O k p k p 3 4 Ωk Ωk log p λ = O k Ω p p 4 k Ωk log p 3 k p between Ωp an Ωk λ = ω Exponential in λ Exponential in λ λ = O Ω log p k p k Ω log p λ = O p k p Ω Ω log p λ = O p Ω 3 p k p k Ω log p between Ω an Ω { } λ = ω in η, +γ Exponential in λ = O ax { λ η, λ γ+ } Exponential in ax { λ η, λ γ+ } k p 4 { } λ = O in η, } } +γ λ = O Ω ax {η, γ+ log p Ω ax {η, γ+ log p k p k 4 Table I: Suary of ain results on parital recovery, an coparisons to the best known necessary conitions for exact recovery. Each entry shows the necessary scaling law for the nuber of saples require to achieve a vanishing error probability. In any of the cases where the necessary conitions for exact recovery lack exponential ters, the corresponing necessary conitions for approxiate recovery are ientical or near-ientical; in particular, see the secon an thir rows corresponing to G k, an the secon an thir rows corresponing to G k,, an the secon row corresponing to G k,,η,γ with = η. While there are logarithic ters issing in soe cases e.g., k vs. k log p, these are typically insignificant in the regies consiere e.g., k = Ωp. 3 In contrast, there are soe cases where significant gaps reain between the best-known conitions for exact recovery an approxiate recovery. The two ost extree cases are as follows: i If k = Θp ɛ for soe sall ɛ > 0, the necessary conitions for G k are Ωp ɛ log p an Ωp +ɛ/, respectively; ii If k = Θp, then the necessary conitions for G k, are Ω log p an Ω log p, respectively. The starting point of our results is a oification of Fano s inequality for the purpose of hanling approxiate recovery. To obtain the above results, we apply this boun to ensebles of graphs that can be broaly classe into two categories. The first consiers graphs with a large nuber of isolate eges, or ore generally, isolate cliques. We characterize how ifficult each graph is to istinguish fro the epty graph, an use this to erive the results given in ite above. On the other han, the results on the exponential ters iscusse in ite arise fro consiering ensebles in which several groups of noes are always highly correlate ue to the presence of a large nuber of eges aong the, thus aking it ifficult to eterine precisely which eges these are. We provie nuerical results on our ensebles in Section VI in orer to support our theoretical finings. Specifically, we ipleent optial or near-optial ecoing rules in a variety of cases, an fin that while partial recovery can be easier than exact recovery, the general behavior of the two is siilar. II. MAIN RESULTS In this section, we present our ain results, naely, algorith-inepenent necessary conitions for the criterion in 4 with all λ ij = λ. Our conitions are written in ters of asyptotic o ters for clarity, but purely non-asyptotic variants can be inferre fro the proofs. Throughout the section, we ake use of the binary entropy function in nats, H θ := θ log θ θ log θ. Here an subsequently, all logariths have base e. As iscusse in Section I-C, all of the bouns presente below contain two ters. For consistency, we always let the first ter correspon to the enseble in which it is ifficult to istinguish each graph fro the epty graph, an we let the secon ter correspon to the ensebles in which several groups of noes exhibit a high correlation, thus aking it ifficult to eterine precisely which eges connect the. All proofs are eferre to later sections; soe preliinary results are presente in Section II, a nuber of ensebles are presente an analyze in Section IV, an the resulting theores are euce in Section V. A. Boune Nuber of Eges Class G k We first consier the class G k of graphs with at ost k eges. It will prove convenient to treat two cases separately epening on how k scales with p. Theore. Class G k with k p/4 For any nuber of eges such that k an k p/4, an any istortion

4 4 level q ax = θk for soe θ 0, 4, it is necessary that { e λ k / log H θ n ax, 6λk log p/ } δ o log p/ k kλ tanh λ θk in orer to have P e q ax δ for all G G k. We procee by consiering two cases as in [7]. In the case that λ k at any rate faster than poly-logarithic in p or even poly-logarithic with a power that is not too sall, the saple coplexity is oinate by the exponential ter e λ k /, an any recovery proceure requires a huge nuber of saples. Thus, in this case, even the approxiate recovery proble is very ifficult. On the other han, if λ = O k then the secon conition in 5 gives a saple coplexity of Ωk log p k, since tanh λ = Oλ as λ 0. Apart fro the slightly ifferent scaling of Ωk log p k in place of Ωk log p, these observations are the sae as those ae for exact recovery in [7], where the best known necessary conitions for G k were given. Thus, we have reache siilar conclusions even allowing for nearly a quarter of the eges to be in error. Theore. Class G k with k = Ωp For any nuber of eges of the for k = cp +ν for constants c > 0 an ν [0,, an any istortion level q ax = θk for soe θ 0, 4, it is necessary that { e λ k / log H θ n ax, 6λk } log H θ δ o λ eλ cosh4λcp ν e λ cosh4λcp ν + in orer to have P e q ax δ for all G G k. As above, the saple coplexity is exponential in λ k ue to the first ter in 6. On the other han, we clai that when λ = O k, the secon ter in 6 leas to the saple coplexity Oin{k, p / k}. To see this, we choose k as in the theore stateent an note that λp ν = Op +ν+ν = Op ν ; since cosh ζ = + Oζ as ζ 0, this iplies that coshcλp ν = + Op ν. We thus have e λ coshcλp ν = + Op +ν + Op ν, which finally yiels 5 6 e λ coshcλp ν e λ coshcλp ν + = Oax{p +ν, p ν } = Oax{/ k, k/p }. When k = Ωp an k = Op 4/3, we have in{k, p / k} = k, an hence, these observations are again the sae as those ae for exact recovery in [7], except that our growth rates o not inclue a log p ter; this logarithic factor is insignificant copare to the leaing ter k = Ωp. In contrast, the gap is ore significant when k p 4/3 ; in the extree case, when k = Θp ɛ for soe sall ɛ > 0, we obtain a scaling of Ωp +ɛ/, as oppose to Ωk log p = Ωp ɛ log p. B. Boune Degree Class G k, Next, we consier the glass G k, of graphs such that every noe has egree at ost, an the total nuber of eges oes not excee k. Theore 3. Class G k, with k p/4 For any axial egree > an nuber of eges k such that k = ω an k p/4, an any istortion level q ax = θk for soe θ 0, 4, it is necessary that { e λ /4 log H n ax θ 3λ, log p/ k log p/ } θk δ o 7 kλ tanh λ in orer to have P e q ax δ for all G G k,. The first ter in 7 reveals that the saple coplexity is exponential in λ. On the other han, if λ = O then the secon ter gives a saple coplexity of Ω log p k. We cannot irectly copare Theore 3 to [7], since there k was assue to be unrestricte for the egree-boune enseble. However, the analysis therein is easily extene to G k,, an oing so recovers the nearly ientical observations to those above, as suarize in Table I. In this sense, Theore 3 atches the best known necessary conitions for exact recovery even when nearly a quarter of the eges ay be in error, up to the replaceent of log p by log p k. Note that by assuption in the theore we have p k 4. Theore 4. Class G k, with k = Ωp For any axial egree > an nuber of eges k such that k = ω an k p for soe, an any istortion level q ax = θk for soe θ 0, 4, it is necessary that { e λ /4 log H n ax θ 3λ, } log H θ δ o λ eλ coshλ e λ coshλ + in orer to have P e q ax δ for all G G k,. Again, the saple coplexity is exponential in λ. By soe stanar asyptotic expansions siilar to those following Theore, we have eλ coshλ e λ coshλ + = O ax {, } whenever λ = O ; hence, the secon conition in 8 becoes n = Ω in {, }. Thus, if = then we again get the esire n = O log p behavior; this eans that we can allow for k up to Op. More generally, we instea get the possibly weaker scaling law n = Ω in {, 3 / }, which is equivalent to n = Ω in { }, 3 p k when k = Θp. In the extree case, when k = Θp the highest growth rate possible given the egree constraint alone, this only recovers Ω log p scaling. C. Sparse Separator Class G k,,η,γ We now consier the class G k,,η,γ of graphs in G k, that satisfy the η, γ-separation conition [0]. We focus on the 8

5 5 case k p/4, since the ain graph enseble that we consier for this class is not suite to the case that k = ωp. Theore 5. Class G k,,η,γ with k p/4 Fix any paraeters, k, η, γ with k p/4 an η, an let be an integer in { 0,..., η}. For any istortion level q ax = θ cη k cηη+γ+ for soe θ 0, an c η, ], it is necessary that { + coshλ cη +tanh λ γ+ tanh λ n ax γ+ λcη log H θ, log p/ k log p/ } q ax δ o kλ tanh λ 9 in orer to have P e q ax δ for all G G k,,η,γ. We procee by consiering only the case λ = O, though siplifications of Theore 5 for λ are also possible. With λ = O, we have coshλ cη = e ζλ cη for soe ζ = Θ, an siilarly +tanh λ γ+ tanh λ = e ζ λ γ+ γ+ for soe ζ = Θ [8, Sec. 5]. These ientities reveal that the saple coplexity is exponential in both λ η an λ γ+. On the other han, if λ = O η an λ = O then γ+ the secon ter in 9 gives n = Ωax{η, γ+ } log p k. Since q ax = cη θ k cηη+γ+, in the case that = / η, we are only in the regie of a constant fraction of errors if γ = Θη. This is true, for exaple, if η = Θ so that the separator set size is a fixe fraction of the axiu egree, an γ = Θ so that the separation is with respect to paths of a boune length. More generally, to hanle larger values of q ax, one can choose a saller value of, thus leaing to a larger value of q ax but a less stringent conition on the nuber of easureents in 9. In the extree case, = 0, an then we are always in the regie of a constant proportion of errors; the isavantage is that this yiels a necessary conition Ωη log p not epening on or γ. The graph faily stuie in [8, Th. ] was soewhat ifferent, in particular not putting any constraints on the axial egree nor the nuber of eges. Nevertheless, by choosing the paraeters in the proof therein to eet these constraints, one again obtains siilar conitions to those above, as suarize in Table I. In particular, for any choice of that grows as Θ, the scaling laws for exact recovery an approxiate recovery coincie. III. AUXILIARY RESULTS Here we provie a nuber of auxiliary results that will be use to prove the theores in Section II. We first present a general for of Fano s epening on both the Kullback-Leibler KL ivergence an Haing istance between graphs, an then provie a nuber of properties of Ising oels that will Specifically, in [8, Sec. 9.], one can set t ν = η to satisfy the egree constraint, an then choose α = k t ν γ++η to ensure there are at ost k eges in total. be useful for characterizing the KL ivergence an Haing istance in specific scenarios. A. Fano s Inequality for Approxiate Recovery As is coon in stuies of algorith-inepenent lower bouns in learning probles, we ake use of bouns base on Fano s inequality [7, Sec..0]. We first briefly outline the ost relevant results for the exact recovery proble. Recall the efinitions of P e an P e q ax in 3 4 with respect to a given graph class G. It is known that for any subset T G, an any covering set C T ɛ such that any graph G T has an ɛ-close graph G C T ɛ satisfying DP G P G ɛ, we have [8] P e log C T ɛ + nɛ + log. 0 log T In particular, if C T ɛ is a singleton, solving for n gives the necessary conition n log T δ log ɛ log T in orer to have P e δ. For approxiate recovery, we consier ensebles i.e., choices of T for which the ecoer s outputs ay lie in soe set T without loss of optiality; in ost cases we will have T = T, but in general, T nee not even be a subset of the graph class G. We use the terinology that the Haing istance between two graphs G = V, E an G = V, E is equal to E E, where E E = E\E E \E. We use the following generalization of. Lea. Suppose that the ecoer iniizing the average error probability with respect to a istortion level q ax, average over a graph uniforly rawn fro a set T G, always outputs a graph in soe set T. Moreover, suppose that there exists a graph G such that DP G P G ɛ for all G T, an that there are at ost Aq ax graphs in T within a Haing istance q ax of any given graph G T. Then it is necessary that n log T log Aq ax δ log ɛ log T in orer to have P e q ax δ. Proof. See Appenix A. B. Properties of Ferroagnetic Ising Moels We will use a nuber of useful results on ferroagnetic Ising oels, each of which is either self-evient or can be foun in [7] or [8]. We start with soe basic properties. Lea. For any graphs G an G with ege sets E an E respectively, we have the following: i For any pair i, j, we have [7] E G [X i X j ] = P G [X i = X j ]. 3

6 6 ii The ivergence between the corresponing istributions satisfies [8, Eq. 4] DP G P G EG [X i X j ] E G [X i X j ] i,j E\E λ + i,j E \E i,j E\E λ + i,j E \E λ E G [X i X j ] E G [X i X j ] EG [X i X j ] 4 λ E G [X i X j ]. 5 iii If E E, then we have for any pair i, j that [8, Eq. 3] E G [X i X j ] E G [X i X j ]. 6 iv Let V,..., V K be a partition of V into K isjoint non-epty subsets. If G an G are such that there are no eges between noes in V i an V j when i j, then DP G P G = K DP Gi P G i, 7 i= where G i = V, E i, with E i containing the eges in E between noes in V i an analogously for G i. The reaining properties concern the probabilities, expectations an ivergences associate with ore specific graphs. Lea 3. i If G is obtaine fro G by reoving a single ege i, j, then [8, Eq. 9] P G [X i = X j ] P G [X i = X j ] = eλ P G [X i = X j ] P G [X i = X j ] an [8, Lea 4] 8 DP G P G λ tanh λ. 9 ii Let G contain a clique on noes an no other eges, an let G be obtaine fro G by reoving a single ege i, j. Then, efining :=, we have [7, Eq. 3] P G [X i = X j ] P G [X i = X j ] j exp λ j exp λj = j=0 j=0 Moreover, we have [7, Lea ] an j exp λ. 0 j P G [X i = X j ] E G [X i X j ] + e λ/ eλ e λ + e λ. iii Suppose that for soe ege i, j E E, there exist at least noe-isjoint paths of length l between i an j in G. Then [8, Lea 3] E G [X i X j ] + +tanh λ l. 3 tanh λ l If the sae is true in both G an G for all i, j E E, then [8, Cor. 3] λ E E DP G P G + +tanh λ l. 4 tanh λ l iv More generally, if there exist at least l noe-isjoint paths of length l l between i, j for l =,..., L, where the values of l l are all istinct, then E G [X i X j ] + L l= +tanh λ l l tanh λ l l l. 5 IV. ENSEMBLES AND THEIR NECESSARY CONDITIONS In this section, we provie necessary conitions for the approxiate recovery of a nuber of ensebles, aking use of the tools fro the previous section. In particular, we seek choices of T, T an Aq ax for substitution into Fano s inequality in Lea. Once these conitions are establishe, it is straightforwar to establish our ain theores; this is one in the Section V. A. Enseble : Many Isolate Eges Suppose that p is even; for o values, the sae conclusions apply by siply consiering an arbitrary subset of size p. This enseble contains nuerous isolate eges, with the iea being that if λ is sall then it is ifficult to eterine precisely which ones are present. It is constructe as follows for soe integer paraeter α p/4: Ensebleα: Group the p vertices into p/ pairs in an arbitrary anner. Each graph in T is obtaine by connecting anywhere between 0 an α of those pairs. We have the following properties: The nuber of graphs is T = α p/ i=0 i p/ α. The axiu egree of each graph is either zero or one. We ay set T to contain the graphs whose eges are a subset of the the p/ above connections; note that this set is larger than T, which only perits up to α eges in total. By a siple counting arguent, the nuber of graphs within a Haing istance q ax of any single graph is upper boune as Aq ax q ax p/ q=0 q + q p/ ax q ax, assuing qax p/4. Fro 9, the KL ivergence fro a single-ege graph to the epty graph is upper boune by λ tanh λ. Hence, an using 7, any graph in T has a KL ivergence to the epty graph of at ost ɛ = αλ tanh λ. Cobining these with gives the necessary conition log p/ α log + q p/ ax q ax n δ log αλ tanh λ log p/ α 6

7 7 for P e q ax δ. Letting q ax = θ α for soe θ 0,, this siplifies to n log p/ α log p/ θ α δ o 7 αλ tanh λ since p/ α. B. Enseble : Many Isolate Groups of Noes This enseble provies an alternative to Enseble that allows for significantly ore eges, in particular allowing for k = ωp. It is constructe as follows with integer paraeters an α: Enseble,α: For α fixe groups of noes, each containing noes. Each graph in T is fore by foring arbitrarily any eges within each group, but no eges between the groups. We have the following: The nuber of noes foring these groups is α. The total nuber of possible eges is α, an hence the total nuber of graphs is T = α. The axial egree of each graph is at ost. The ecoer can output an eleent of T without loss of optiality, since any inter-group eges eclare to be present are guarantee to be wrong. Thus, we ay set T = T. The nuber of graphs within a Haing istance q ax of any single graph is Aq ax = q ax α q=0 q + q α ax q ax, assuing qax α. In Lea 4 below, we show that the KL ivergence of the graph associate with one group to the corresponing epty graph is upper boune by λ e λ coshλ e λ coshλ+. Hence, the KL ivergence of any G T to the epty graph is upper boune by ɛ = α λ e λ coshλ e λ coshλ+ ue to 7. Substituting these into, setting q ax = θ α for soe θ 0,, an applying soe siplifications, we obtain the following necessary conition for P e q ax δ: n log H θ λ eλ coshλ δ o e λ coshλ+ 8 whenever α. Note that the binary entropy function arises fro the ientity N θn = e nh θ+o as N. Lea 4. Let G enote an arbitrary graph with eges connecte to at ost noes, an let G be the epty graph. Then, it hols that DP G P G λ eλ coshλ e λ coshλ +. 9 Proof. We prove the clai for the case that G contains a single -clique; the general case then follows in a siilar fashion using 6. Let G be obtaine fro G by reoving a single ege, say inexe by i, j. Defining qg := P G [X i = X j ] an :=, we have fro 8 that an fro 0 that qg qg = j=0 qg qg = qg eλ qg, 30 j exp λ j exp λj j=0 j exp λ. j 3 Noting the syetry of the suans with respect to j an j, we obtain the following when is o the case that is even is hanle siilarly, leaing to the sae conclusion: qg qg = / j=0 j exp λ j cosh λj / j=0 j exp λ j 3 ax cosh λj 33 j=0,..., / = cosh λ 34 cosh λ 35 Substituting 35 into 30, solving for qg, an converting fro probability to expectation via 3, we obtain E G [X i X j ] eλ coshλ e λ coshλ The proof is conclue by substituting into 4 an noting that E G [X i X j ] = 0, E\E =, an E \E = 0. C. Enseble 3: Large Inter-Connecte Cliques This enseble involves cliques with nuerous eges between the, with the iea being that it is ifficult to eterine precisely which inter-clique connections are present, particularly for large cliques an large values of λ. It is constructe as follows with integer paraeters an α: Enseble3,α: Take an arbitrary subset of the p vertices of size. Construct a fixe builing block as follows: Split the vertices into two sets of size each, fully connect each of those sets, an then put extra eges between the two sets in a fixe but arbitrary one-to-one fashion. For α isjoint copies of this builing block to obtain a base graph G 0. Each graph in T is fore by taking G 0 an aing an arbitrary nuber of aitional eges between each pair of partially-connecte cliques. Thus, G 0 itself contains the fewest eges within T, an the union of α cliques of size contains the ost eges. An illustration of one builing block is given in Figure. We have the following: The nuber of noes foring these groups is α, an the nuber of eges in each graph is upper boune by α α.

8 8 inter-clique connections -cliques Figure : Builing block for Enseble 3 with = 4. The nuber of potential eges between two -cliques is, an of the are always there in each builing block. Hence, the nuber of ways of aing eges to one builing block is, an the total nuber of graphs is α. The axial egree of each graph is at ost. Siilarly to Enseble, the ecoer can output an eleent of T without loss of optiality, so that T = T. The nuber of graphs within a Haing istance q ax of any single graph is Aq ax = q ax α q=0 q + q α ax q ax, assuing qax α. In Lea 5 below, we show that the KL ivergence of each builing block to the -clique graph is upper boune by λ 4 e λ /. Thus, the KL ivergence fro any G T to the union of α -cliques is upper boune by ɛ = λα 4 e λ / ue to 7. Substituting these into, setting q ax = θ 3 α for soe θ 3 0,, an siplifying, we obtain n eλ / log H θ 3 δ o λ 37 whenever α. Lea 5. Let G enote any single builing block of the above for obtaine by foring two cliques of size an connecting eges between the, an let G be the corresponing graph containing a -clique. Then DP G P G λ 4 e λ /. 38 Proof. Fro 6 an, we have for any i, j within either of the two -cliques that P G [X i = X j ] 39 + e λ/, 40 + eλ/ where :=. Thus, by the union boun, the probability that each of the cliques have noes that all take the sae value satisfies P G [all noes sae within each clique] + e. λ/ 4 Next, we consier the probabilities of the two cliques taking a coon value vs. two ifferent values. Letting A ν,σ be the event that the ν-th clique has values all equal to σ {+, }, we have fro that P G [A,+ A,+ ] = λ Z exp P G [A,+ A, ] = λ Z exp Taking the ratio between the two gives P G [A,+ A,+ ] P G [A,+ A, ] = eλ. 44 By the sae arguent, this is also the ratio between any analogous events with the sae signs in the nuerator an iffering signs in the enoinator. The sae also hols true when conitioning on each of the two cliques having coonvalue noes; in this case, the left-han sie of 44 takes the for p p, an solving for p yiels P G [all noes sae all noes sae within each clique] =, 45 + eλ where all noes refers to the noes aking up the two cliques. Multiplying this with 4 gives P G [all noes sae] + e. 46 λ/ + eλ Hence, an using 3, we have for all i, j even in ifferent cliques that 4 E G [X i X j ] + e. 47 λ/ + eλ Finally, the nuber of eges that are in the coplete graph G but not in G is trivially upper boune by, an thus substitution into 5 yiels DP G P G λ 4 + e + λ / + e λ. 48 The proof is conclue by writing 4 + e + λ / + e λ 4 e + λ / eλ eλ / 50 Reark. In this enseble, there are α eges known with certainty, an a possible further α that are unknown. Thus, slightly ore than half of the potential eges are known. This liits the values of q ax that are eaningful when applying this enseble, an is the reason for the constraint q ax k/4 in the Theores 4. However, one can generalize this enseble by consiering ore than two groups of -cliques such that each pair has inter-clique connections. With this extension, the fraction of potential eges that are known can be ae arbitrarily close to zero, an siilar results to those shown in Table I for G k respectively, G k, can be obtaine even when q ax = θ k repectively, q ax = θ k for soe θ 0,

9 9 D. Enseble 4: Many Noe-Disjoint Paths This enseble is base on foring a large nuber of noe-isjoint paths between pairs of noes, with the iea being that it is ifficult to eterine whether or not irect eges also exist between those noes [8]. It is constructe as follows for soe integers η, η,, l, α: Enseble4η,η,,l,α: Take an arbitrary subset of the p vertices of size η an label the,,..., η. For each consecutive pair of these noes incluing the wrappe-aroun pair η,, for η noe-isjoint paths of length two between the, an also for noe-isjoint paths of length l between the. For a base graph G 0 by taking α copies of this graph. Each graph in T is fore by taking G 0 an aing arbitrarily any eges aong the η center noes of each builing block. Thus, G 0 itself has the fewest eges, whereas the graph with α η aitional center eges contains the ost eges. An illustration of one builing block is shown in Figure. noes length-` paths length- paths Figure : Builing block for Enseble 4 with η = 5, η =, =, an l = 3. We have the following: The nuber of noes within each builing block is η + η + l, an hence the total nuber of noes is αη + η + l. Within each builing block, there are up to η eges in the center, as well as η η further eges foring paths of length two, an η l eges foring paths of length l. Hence, the total nuber of eges is between αη η + l an αη η / + η + l. The total nuber of graphs is T = αη. The axial egree is less than η + η +. Siilarly to Ensebles an 3, we ay set T = T. The nuber of graphs within a Haing istance q ax of any given graph is Aq ax = q ax α η q=0 q + q α η ax q ax, assuing qax α η. Using Lea 6 below, along with 7, the KL ivergence fro any graph in T to the corresponing graph with all centers connecte is upper boune by λαη ɛ = η η +tanh λ l. + coshλ tanh λ l Substituting these into an setting q ax = θ 4 α η for soe θ 4 0, gives n + coshλ η l +tanh λ tanh λ l log H θ 4 λη provie that α η. δ o 5 Lea 6. Let G enote any construction of the above builing block, an let G be the corresponing builing block with all of the center noes connecte. Then λη η DP G P G + coshλ η +tanh λ l. 5 tanh λ l Proof. We know fro 5 that the joint istribution between any two consecutive noes in the center satisfies E G [X i X j ] + coshλ η +tanh λ l, 53 tanh λ l since +tanh λ tanh λ = coshλ. Using 3, this iplies P G [X i = X j ] + coshλ η +tanh λ l. 54 tanh λ l Thus, by the union boun, the probability that all η of the center noes take the sae value satisfies P G [all center noes sae] η + coshλ η +tanh λ l tanh λ l. 55 Again using 3, this iplies for any pair of center noes i, j even non-consecutive that η E G [X i X j ] + coshλ η +tanh λ l. 56 tanh λ l Observing that the corresponing ege sets E an E satisfy E \E η an E\E = 0, 5 follows fro 5. V. APPLICATIONS TO GRAPH FAMILIES Finally, we prove our ain results by applying the ensebles fro the previous section to the graph failies introuce in Section I-A. All of the necessary conitions on n state in this section are those neee to obtain P e q ax δ, where the graph class efining P e will be clear fro the context. A. Proofs of Theores : Boune Eges Enseble For the class G k of graphs with at ost k eges, we have the following: If k p/4, then using Enseble with α = k, we obtain fro 7 that n log p/ k log p/ θ k δ o 57 kλ tanh λ

10 0 provie that q ax θ k for soe θ 0,. If k = cp +ν for soe c > 0 an ν [0,, then we use Enseble with = cp ν an α = p/ = c p ν + o chosen so that α p noes are use in the construction. The nuber of possible eges is α α p cp+ν, as esire. We obtain fro 8 that n log H θ λ eλ coshλcp ν δ o 58 e λ coshλcp ν + provie that q ax θ α for soe θ 0,. Substituting the choices of an α into the latter expression, we fin that q ax can be as large as θ k + o. Using Enseble 3 with α = an = k/ chosen so that the nuber of eges oes not excee α k, we obtain fro 37 along with the ientity that n eλ k/ / log H θ 3 δ o 59 6λk provie that q ax θ 3 α for soe θ 3 0,. Substituting the choices of an α into the latter k expression, we fin that q ax can be as large as θ 3 + o, provie that k. Note that this construction uses α k noes, which is asyptotically less than p since k = op. We obtain Theore fro 57 an 59, an Theore fro 58 an 59. Specifically, we set q ax = θk for soe θ 0, 4, an by equating this with the above upper bouns on q ax we see that we ay set θ = θ, θ = θ + o an θ 3 = θ + o. B. Proofs of Theores 3 4: Boune Degree Enseble For the class G k, of graphs such that every noe has egree at ost, an the total nuber of eges oes not excee k, we have the following: If k p/4, then using Enseble with α = k, we obtain fro 7 that n log p/ k log p/ θ k δ o 60 kλ tanh λ provie that q ax θ k for soe θ 0,. If k = Ωp, then using Enseble with = chosen so that the axial egree oes not excee an α = k/ chosen so that the nuber of eges α oes not excee k, we obtain fro 8 that n log H θ λ eλ coshλ e λ coshλ + δ o 6 whenever q ax θ α for soe θ 0,. Substituting the choice of α, we fin that q ax can be a large as θ k+o. Note also that the nuber of noes use is upper boune as α k = k, which is upper boune by p provie that k p. Using Enseble 3 with = / chosen so that each block has noes with egree not exceeing an α = k hence ensuring that the nuber of eges oes not excee α k, we obtain fro 37 that n eλ /4 log H θ 3 δ o. 6 3λ when q ax θ 3 α for soe θ 3 0,. Substituting the choice of α to obtain α = k + o, an then writing / + /, we fin that the latter conition hols provie that q ax / θ 3 k + o. The nuber of noes use k is α = k k, which is upper boune by p provie that k p. We obtain Theore 3 fro 60 an 6, an Theore 4 fro 6 an 6. Siilarly to the previous subsection, we set θ = θ, θ = θ + o, an θ 3 = θ + o. C. Proofs of Theore 5: Sparse Separator Enseble For the class G k,,η,γ following: cf. Section II-C, we have the If k p/4, then again using Enseble with α = k, we obtain fro 6 that n log p/ k log p/ q ax δ o, 63 kλ tanh λ where we recall the choice q ax = θ α therein. Using Enseble 4 with η = cη for soe c η, ], η = cη, l = γ + chosen to ensure that the η, γ-separation conition is satisfie, / η chosen so that the axial egree is upper boune by η + η + η +, an α = k cηcη/+ cη+γ+ chosen to ensure the total nuber of eges αη η / + η + l oes not excee k, we obtain fro 5 that n + coshλ cη +tanh λ γ+ tanh λ γ+ λcη log H θ 4 δ o 64 provie that q ax θ 4 αcη / for soe θ 4 0,. Here we have use ζ ζ ζ an cη cη /. Note that the graph in this enseble with the ost eges has at least as any eges as noes, since each noe is connecte to at least two eges. Thus, since we have assue k p/4 an we have alreay chosen the paraeters to ensure there are at ost k eges, we have also ensure that less than p noes are use. Substituting the above choice of α into the upper boun on q ax, we fin that q ax can be as large as cη k θ 4 cηcη/ + cη + γ + cη k θ 4, 65 cηη + γ + since cη/ + cη η for c [0, ]. We obtain Theore 5 by cobining 63, 64 an 65, an renaing θ 4 as θ. Choosing c in this range ensures that η.

11 VI. NUMERICAL RESULTS In this section, we nuerically siulate the graph learning proble for Ensebles, 3 an 4 in Section IV, as well as the analogous ensebles use for exact recovery in [7], [8]; we oit Enseble since it is an extension of Enseble, but is less suitable for coparison to ensebles fro the existing literature. Before proceeing, we iscuss the optial ecoing techniques of the two recovery criteria. Suppose that the graph G is uniforly rawn fro soe class G. In the case of exact recovery, the optial ecoer uses the axiu-likelihoo ML rule Ĝ = arg ax P G [X], 66 G G where P G [X] is the probability of observing the saples X {0, } n p when the true graph is G. In contrast, the optial rule for partial recovery is Ĝ = arg ax G G G : E E q ax P G [X], 67 where E an E are the ege sets of G an G respectively. Both 66 an 67 are, in general, coputationally intractable, requiring a search over the entire space G. However, in the exaples below, we are able to nuerically siulate 66 by using various tricks such as syetry arguents. While we nee to consier relatively sall graph sizes for Ensebles 3 an 4, these will still be aequate for generating results that support the theory. Unfortunately, we foun the ipleentation of 67 uch ore ifficult, an we therefore also use 66 for partial recovery even though, in general, it is only optial for exact recovery. Nevertheless, even with partial recovery, we expect ML to provie a benchark that that is unlikely to be beaten by any practical ethos. In all of the experients, the error probabilities are obtaine by evaluating the epirical average over 5000 experients. A. Enseble an a Counterpart fro [7] It is shown in [7] that if one consiers all graphs with a single ege, then it is ifficult to istinguish each of these fro the epty graph if λ is sall, thus aking exact recovery ifficult. In Figure 3, we siulate the perforance of this enseble with p = 00. Since the partition function Z see is the sae for all graphs in this enseble, the ML rule 66 siply aounts to eclaring the single ege to be the pair i, j aong the p possibilities such that Xi = X j in the highest nuber of saples. For coparison, we plot the partial recovery error probability for Enseble with p = 00 an α =, setting q ax = 3 so that up to a quarter of the eges ay be in error. The axiu-likelihoo rule 66 is ipleente as follows: For each l = 0,..., α, let Ĝ l be the ost likely graph aong those having l eges. Since all such graphs have the sae partition function, this aounts to choosing the l eges such that the two corresponing noes agree in as any observations as possible, which is easily ipleente via sorting. Error Probability λ = λ = 0.5 λ = Nuber of Measureents Figure 3: Epirical perforance for Enseble partial recovery; re bol an its counterpart fro [7] exact recovery; blue non-bol. Aong the graphs in {Ĝ0,..., Ĝα}, return the one with the highest likelihoo. Note that this etho requires is to copute only α + likelihoo scores, rather than p α. As preicte by our theory, the general behavior of the error probability as a function of n is siilar in the above two scenarios. Moving to partial recovery oes provie soe benefit, but it appears to be only in the constant factors. More specifically, across the range shown, the nuber of easureents require to achieve a given error probability in [0.0, 0.5] iffers for the two ensebles an recovery criteria only by a ultiplicative factor in the range [,.]. In both cases, the learning proble becoes increasingly ifficult as λ becoes saller, since the eges are weaker an therefore ore ifficult to etect. B. Enseble 3 an a Counterpart fro [7] A counterpart to Enseble 3 fro [7] consiers the possible graphs on noes obtaine by reoving a single ege fro the -clique. Thus, every graph is ifficult to istinguish fro the -clique, particularly as an λ increase, an exact recovery is ifficult. In Figure 4, we plot the perforance of this enseble with = 8. In this case, ML ecoing aounts to choosing the pair i, j such that X i X j in the highest nuber of saples. For coparison, we consier Enseble 3 with = 4 an α =, chosen so that the axial nuber of eges an egree atch those of the enseble fro [7] with = 8. We set q ax = 3, so that up to a quarter of the unknown eges ay be in error. We perfor ML ecoing using a brute force search over the possible graphs. Copare to the previous exaple, the gap between the curves for partial recovery an exact recovery are ore significant. This is because although both our results an those of [7] prove that the saple coplexity is exponential in λ, the exponent in [7] is ouble that of ours. Intuitively, this is because we work with cliques of half the size. Despite this, the

12 λ = Error Probability 0 - λ = 0. λ = 0.5 λ = 0.75 Error Probability 0 - λ = 0. λ = Nuber of Measureents Figure 4: Epirical perforance for Enseble 3 partial recovery; re bol an its counterpart fro [7] exact recovery; blue non-bol. general behavior of our curves an those of [7] is siilar, with the saple coplexity rapily growing large as λ increases ue to higher correlations aong the 8 noes. C. Enseble 4 an a Counterpart fro [8] A counterpart to Enseble 4 fro [8] first constructs α isjoint builing blocks, each of which connects two noes i, j an then further fors η noe-isjoint paths of length between the. Each graph in the enseble is then obtaine by reoving the irect ege while leaving the length- paths unchange fro one of the α builing blocks. We consier this construction with α = 4 an η = 8, thus leaing to the use of p = 40 noes an k = 68 eges, an a axial egree = 9. Figure 5 plots the perforance of the ML ecoer, which aounts to counting the nuber agreeents between the α pairs of central noes one per builing block, an eclaring the ege to be absent in the one with the ost isagreeents. For coparison, we consier Enseble 4 with η = 4, η = 3, = 0 an α = ; this construction uses p = 3 noes an k = 60 eges, an has a axial egree = 9, thus being coparable to the above construction fro [8]. We set q ax = 3, so that up to a quarter of the unknown eges ay be in error. We perfor ML ecoing using a brute force search over the possible graphs, which siplifies to perforing ML separately on the 6 possible graphs corresponing to each of the two builing blocks. Once again, we observe the sae general behavior between our enseble an that of [8]. While it ay appear unusual that the exact recovery curves have a saller error probability at low values of n, this occurs because even a rano guess achieves an probability of exact recovery of 4 for the enseble in [8] with α = 4. Despite this, we see partial recovery is easier for large n as expecte, an that in both cases the recovery proble rapily becoes ore ifficult as λ increases ue to higher correlations aong the noes Nuber of Measureents Figure 5: Epirical perforance for Enseble 4 partial recovery; re bol an its counterpart fro [7] exact recovery; blue non-bol. VII. CONCLUSION We have provie inforation-theoretic lower bouns on Ising oel selection with approxiate recovery for a variety of graph classes. For a wie range of scaling regies of the relevant paraeters, we have obtaine necessary conitions with the sae scaling laws as the best known conitions for exact recovery, thus inicating that approxiate recovery is not uch easier. To this en, we presente a generalize for of Fano s inequality for hanling approxiate recovery, an applie it to a variety of graph ensebles. These were broaly categorize into those where it is ifficult to istinguish each graph fro the epty graph, an those where it is ifficult to eterine which eges between highly-correlate groups of noes are present. In both cases, we require a significant eparture fro the ensebles consiere for exact recovery [7], [8] in which the graphs iffer in only one or two eges. Since our analysis is base on constructing ensebles of graphs having a KL ivergence that is close to a single graph, one ay expect that uner a recovery criterion base on DP G PĜ being sall, there is uch ore to be gaine. Other irections for further work inclue oels beyon the Ising oel e.g., non-binary, Gaussian, an stuies of achieving approxiate recovery with practical algoriths. APPENDIX A PROOF OF LEMMA The proof follows stanar steps in the erivation of Fano s inequality as in [8], but with suitable oifications to hanle the approxiate recovery criterion; see [8] for analogous oifications in the context of support recovery. Let G be uniforly istribute on T, let Ĝ be the estiate of G, an let E an Ê be the corresponing ege sets. Moreover, let P e q ax be the error probability P[ E E > q ax ] average over the rano graph G. By assuption, we ay consier ecoers such that Ĝ T without loss of optiality. Defining the error inicator E := { E E > q ax } an applying the chain rule for entropy

13 3 in two ifferent ways, we have HE, G Ĝ = HG Ĝ + HE G, Ĝ 68 = HE Ĝ + HG E, Ĝ. 69 We have HE G, Ĝ = 0 since E is a function of G, Ĝ, an HE Ĝ log since E is binary. Moreover, we have HG E, Ĝ = P eq ax HG E = 0, Ĝ + P e q ax HG E =, Ĝ 70 P e q ax log Aq ax + P e q ax log T, 7 where 7 follows fro the efinition of A ax in the lea stateent an the fact that E = 0 iplies that G is within a istance q ax of Ĝ, an we have use the fact that the entropy is upper boune by the logarith of the nuber of eleents of the support. We have now hanle three of the ters in 68 69, an for the final one we write HG Ĝ = IG; Ĝ + HG = IG; Ĝ + log T, since G is unifor on T. Substituting the preceing observations into an perforing soe siple rearrangeents gives P e q ax IG; Ĝ + log log T log Aq ax. 7 Finally, we boun the utual inforation using the steps of [8], which are state here without the etails in orer to avoi repetition: We use the ata processing inequality to write IG; Ĝ IG; X where X contains the n inepenent saples fro P G, an then use a covering arguent, as well as the assuption containing G in the lea stateent, to write IG; X nɛ. Substituting into 7, solving for n, an writing P e q ax P e q ax, we obtain the esire result. REFERENCES [] S. Gean an D. Gean, Stochastic relaxation, Gibbs istributions, an the Bayesian restoration of iages, IEEE Trans. Patt. An. Mach. Int., no. 6, pp. 7 74, 984. [] R. J. Glauber, Tie-epenent statistics of the Ising oel, J. Math. Phys., vol. 4, no., pp , 963. [3] R. Durbin, S. R. Ey, A. Krogh, an G. Mitchison, Biological sequence analysis: Probabilistic oels of proteins an nucleic acis. Cabrige Univ. Pressn, 998. [4] C. D. Manning an H. Schütze, Founations of statistical natural language processing. MIT press, 999. [5] S. Wasseran an K. Faust, Social network analysis: Methos an applications. Cabrige Univ. Press, 994, vol. 8. [6] A. Boganov, E. Mossel, an S. Vahan, The coplexity of istinguishing Markov rano fiels, in Appr. Ran. Cob. Opt. Alg. Tech. Springer, 008, pp [7] N. Santhana an M. Wainwright, Inforation-theoretic liits of selecting binary graphical oels in high iensions, IEEE Trans. Inf. Theory, vol. 58, no. 7, pp , July 0. [8] K. Shanuga, R. Tanon, A. Diakis, an P. Ravikuar, On the inforation theoretic liits of learning Ising oels, in Av. Neur. Inf. Proc. Sys. NIPS, 04. [9] E. Ising, Beitrag zur theorie es ferroagnetisus, Zeitschrift für Physik A Harons an Nuclei, vol. 3, no., pp , 95. [0] A. Anankuar, V. Y. F. Tan, F. Huang, an A. S. Willsky, Highiensional structure estiation in Ising oels: Local separation criterion, Ann. Stats., vol. 40, no. 3, pp , 0. [], High-iensional Gaussian graphical oel selection: Walk suability an local separation criterion, J. Mach. Learn. Res., vol. 3, pp , 0. [] G. Bresler, E. Mossel, an A. Sly, Reconstruction of Markov rano fiels fro saples: Soe observations an algoriths, in Appr., Ran. an Cob. Opt. Algoriths an Techniques. Springer Berlin Heielberg, 008, pp [3] R. Wu, R. Srikant, an J. Ni, Learning loosely connecte Markov rano fiels, Stoch. Sys., vol. 3, no., pp , 03. [4] A. Jalali, C. C. Johnson, an P. K. Ravikuar, On learning iscrete graphical oels using greey ethos, in Av. Neur. Inf. Proc. Sys. NIPS, 0. [5] A. Ray, S. Sanghavi, an S. Shakkottai, Greey learning of graphical oels with sall girth, in Allteron Conf. Co., Control, an Cop., 0. [6] G. Bresler, D. Gaarnik, an D. Shah, Structure learning of antiferroagnetic Ising oels, in Av. Neur. Inf. Proc. Sys. NIPS, 04. [7] G. Bresler, Efficiently learning Ising oels on arbitrary graphs, in ACM Syp. Theory Cop. STOC, 05. [8] P. Ravikuar, M. J. Wainwright, J. D. Lafferty, an B. Yu, Highiensional Ising oel selection using l -regularize logistic regression, Ann. Stats., vol. 38, no. 3, pp , 00. [9] E. Yang, A. C. Lozano, an P. K. Ravikuar, Eleentary estiators for graphical oels, in Av. Neur. Inf. Proc. Sys. NIPS, 04, pp [0] A. Montanari an J. A. Pereira, Which graphical oels are ifficult to learn? in Av. Neur. Inf. Proc. Sys. NIPS. Curran Associates, Inc., 009. [] R. Tanon an P. Ravikuar, On the ifficulty of learning power law graphical oels, in IEEE Int. Syp. Inf. Theory, 03. [] N. Meinshausen an P. Bühlann, High-iensional graphs an variable selection with the Lasso, Ann. Stats., vol. 34, no. 3, pp , June 006. [3] W. Wang, M. Wainwright, an K. Rachanran, Inforation-theoretic bouns on oel selection for Gaussian Markov rano fiels, in IEEE Int. Syp. Inf. Theory, 00. [4] V. Jog an P.-L. Loh, On oel isspecification an KL separation for Gaussian graphical oels, 05, [5] P. Ravikuar, M. J. Wainwright, G. Raskutti, B. Yu et al., Highiensional covariance estiation by iniizing l -penalize logeterinant ivergence, Elec. J. Stats., vol. 5, pp , 0. [6] D. Vats an J. M. Moura, Necessary conitions for consistent set-base graphical oel selection, in IEEE Int. Syp. Inf. Theory, 0, pp [7] T. M. Cover an J. A. Thoas, Eleents of Inforation Theory. John Wiley & Sons, Inc., 00. [8] G. Reeves an M. Gastpar, Approxiate sparsity pattern recovery: Inforation-theoretic lower bouns, IEEE Trans. Inf. Theory, vol. 59, no. 6, pp , June 03.