Convex optimization for the planted k-disjoint-clique problem

Transcription

1 Convex optimization for the planted k-dijoint-clique problem Brendan P.W. Ame Stephen A. Vavai arxiv: v4 [math.oc] 2 Dec 203 December 3, 203 We conider the k-dijoint-clique problem. The input i an undirected graph G in which the node repreent data item, and edge indicate a imilarity between the correponding item. The problem i to find within the graph k dijoint clique that cover the maximum number of node of G. Thi problem may be undertood a a general way to poe the claical clutering problem. In clutering, one i given data item and a ditance function, and one wihe to partition the data into dijoint cluter of data item, uch that the item in each cluter are cloe to each other. Our formulation additionally allow noie node to be preent in the input data that are not part of any of the clique. The k-dijoint-clique problem i NP-hard, but we how that a convex relaxation can olve it in polynomial time for input intance contructed in a certain way. The input intance for which our algorithm find the optimal olution conit of k dijoint large clique called planted clique ) that are then obcured by noie edge inerted either at random or by an adverary, a well a additional node not belonging to any of the k planted clique. Introduction Given a et of data, clutering eek to partition the data into et of imilar object. Thee ubet are called cluter, and the goal i to find a few large cluter covering a much of the data a poible. Clutering play a ignificant role in a wide range of application; including, but not limited to, information retrieval, pattern recognition, computational biology, and image proceing. For a recent urvey of clutering technique and algorithm with a particular focu on application in data mining ee [3]. In thi paper, we conider the following graph-baed repreentation of data. Given a et of data where each pair of object i known to be imilar or diimilar, we conider the graph G = V, E) where the object in the given data et are the et of node of G and any two Supported in part by a Dicovery Grant and Potgraduate Scholarhip Doctoral) from NSERC Natural Science and Engineering Reearch Council of Canada), MITACS Mathematic of Information Technology and Complex Sytem), and the US Air Force Office of Scientific Reearch. Intitute for Mathematic and it Application, College of Science and Engineering, Univerity of Minneota, 207 Church Street SE, 400 Lind Hall, Minneapoli, Minneota, 55455, U.S.A., bpame@gmail.com Department of Combinatoric and Optimization, Univerity of Waterloo, 200 Univerity Avenue W., Waterloo, Ontario N2L 3G, Canada, vavai@math.uwaterloo.ca

2 node are adjacent if and only if their correponding object are imilar. Hence, for thi repreentation of the data, clutering i equivalent to partitioning G into dijoint clique. Therefore, for any integer k, the problem of identifying k cluter in the data containing the maximum number of object i equivalent to the maximum node k-dijoint-clique problem of the correponding graph G. Given an undirected graph G = V, E) and integer k [, V ], the maximum node k-dijoint-clique problem refer to the problem of finding the ubgraph K of G compoed of a collection of k dijoint clique, called a k-dijoint-clique ubgraph, maximizing the number of node in K. Unfortunately, ince the k = cae i exactly the maximum clique problem, well-known to be NP-hard [0], the maximum node k-dijointclique problem i NP-hard. In Section 2, we relax the maximum node k-dijoint-clique problem to a emidefinite program. We how that thi convex relaxation can recover the exact olution in two cae. In the firt cae, preented in Section 3, the input graph i contructed determinitically a follow. The input graph conit of k dijoint clique C,..., C k, each of ize at leat ˆr, plu a number of diverionary node and edge inerted by an adverary. We how that the algorithm can tolerate up to Oˆr 2 ) diverionary edge and node provided that, for each i =,..., k, each node in the clique C i i adjacent to at mot Omin{ C i, C j }) node in the clique C j for each j =,..., k uch that i j. In Section 4, we uppoe that the graph contain a k-dijoint-clique ubgraph K and ome additional node, and the remaining nonclique edge are added to the graph independently at random with fixed probability p. We give a general formula for clique ize that can be recovered by the algorithm; for example, if the graph contain N node total and N /4 planted clique each of ize ΩN /2 ), then the convex relaxation will find them. We develop the neceary optimality and uniquene theorem in Section 2 and provide the neceary background on random matrice in Section 4.. The rationale for thi line of analyi i that in real-world application of clutering, it i often the cae that the ought-after cluter are preent in the input data but are hidden by the preence of noiy data. Therefore, it i of interet to find cae of clutering data in which the cluter are hidden by noie and yet can till be found in polynomial time. Our analyi i related in an indirect manner to work on meauring cluterability of data, e.g., Otrovky et al. [6]. In that work, the author prove that a certain clutering algorithm work well if the data ha k good cluter. Our aumption and analyi, however, differ ubtantially from [6] for example, we do not require all the data item to be placed in cluter), o there i no direct relationhip between our reult and their. Our reult and technique can be een a an extenion of thoe in [2] from the maximum clique problem to the maximum node k-dijoint-clique problem. Indeed, in the k = cae, our reult agree with thoe preented in [2], a well a thoe found in earlier work by Alon et al. [], and by Feige and Krauthgamer [8]. Recent paper by Oymak and Haibi [7] and Jalali et al. [4] written ubequently to the initial prepublication releae of thi paper extend our reult to a more general model for clutered data. Specifically, in [7] and [4] the author independently propoe new heuritic for partitioning an input graph G into denely connected ubgraph baed on the heuritic for decompoition of a matrix into low-rank and pare component conidered in [7] and [5]. Under certain aumption on the input graph, Oymak and Haibi [7] and Jalali et al. [4] how that thi approach uccefully recover the correct partition of the graph G into denely connected ubgraph. In particular, the 2

3 correct partition i recovered when the input graph i compoed of everal dijoint clique obcured by noie in the form of random edge addition and deletion. It i important to note that our approach can only tolerate noie in the form of edge addition. The lower bound on the minimum clique ize enuring exact recovery provided by [7] and [4] i equal to that provided in Section 4, although the condition enuring exact recovery provided in [7] and [4] do not require any explicit upper bound on the number of planted clique to be atified. On the other hand, exact recovery i guaranteed for our approach in the precence of up to Oˆr 2 ) diverionary node, while [7] and [4] do not conider noie of thi form. More generally, our reult follow in the pirit of everal recent paper, in particular Recht et al. [8] and Candè and Recht [6], which conider nuclear norm minimization, a pecial cae of emidefinite programming, a a convex relaxation of matrix rank minimization. Matrix rank minimization refer to the problem of finding a minimum rank olution of a given linear ytem. Thee paper have reult of the following general form. Suppoe that it i known that the contraint of the given linear ytem are random in ome ene and that it i known that a olution of very low rank exit. Then the nuclear norm relaxation recover the unique) olution of minimum rank. We will argue that, in the cae that the graph G contain a planted k-dijoint-clique ubgraph K and not too many diverionary edge, a rank k olution, correponding to the adjacency matrix of K, of a ytem of linear equation defined by the input graph G can be recovered exactly by olving a emidefinite program. Like many of the paper mentioned in the previou paragraph, the proof that the convex relaxation exactly recover the combinatorial olution contruct multiplier to etablih that the combinatorial olution atifie the KKT optimality condition of the convex problem. Herein we introduce a new technical method for the contruction of multiplier. In [2], the multiplier are contructed according to imple formula becaue of the fairly imple nature of the problem. On the other hand, in [6], the multiplier are contructed by projection i.e., olving a linear leat-quare problem), which entail a quite difficult analyi. Thi paper introduce a technique of intermediate complexity: we contruct the multiplier a the olution to a ytem of invertible linear equation. We how that the equation are within a certain norm ditance of much impler diagonal plu rank-one) linear equation. Finally, the reult i obtained from tandard bound on the perturbation of the olution of a linear ytem due to perturbation in it coefficient. 2 The Maximum Node k-dijoint-clique Problem Let G = V, E) be a imple graph. The maximum node k-dijoint-clique problem focue on finding k dijoint clique in G uch that the total number of node in thee clique i maximized. We call a ubgraph of G compoed of k dijoint clique a k-dijoint-clique ubgraph. Thi problem i clearly NP-hard ince it i equivalent to the maximum clique problem in the cae that k =. The problem of maximizing the number of node in a k-dijoint-clique ubgraph of G can 3

4 be formulated a the following combinatorial optimization problem max S={v,...,v k } k vi T e ) i=.t. v T i v j = 0, i, j =,..., k, i j 2) [v i v T i ] uv = 0, if uv / E, u v, i =,..., k 3) v i {0, } V, i =,..., k. 4) Here e denote the all-one vector in R N. A feaible olution S = {v,..., v k } for ) i the collection of characteritic vector of a et of dijoint clique of G. Indeed, the contraint 2) enure that the et of node indexed by S are dijoint and the contraint 3) enure that the et of node indexed by S induce complete ubgraph of G and, hence, are clique of G. Note that a feaible olution S = {v,..., v k } need not define a partition of V. That i, a feaible olution need not correpond to a k-dijoint-clique ubgraph of G that contain every node in V. Unfortunately, finding the olution to a nonlinear program with binary contraint i NP-hard in general. The formulation ) may be relaxed to the rank contrained emidefinite program max N N i= j= X ij.t. Xe e, X ij = 0, i, j) / E.t. i j rank X) = k, X 0 5) where N = V, X i an N N real ymmetric matrix, and the notation X 0 mean that X i poitive emidefinite. To ee that 5) i a relaxation of ), uppoe that {C,..., C k } V define a k-dijoint-clique ubgraph of G. Let S = {v,..., v k } be the et of characteritic vector of {C,..., C k }. The matrix X = k i= v i v T i C i 6) i poitive emidefinite with rank equal to k. Note that k i= v i e ince v,..., v k are orthogonal binary vector. It follow that ince v T i e = C i. Moreover, Xe = k i= ) vi vi T e = C i k v i e 7) i= N N X ij = e T Xe = i= j= k vi T e. 8) i= Therefore, every feaible olution S for ) define a feaible olution of 5) with objective value equal to the number of node in the k-dijoint-clique ubgraph defined by S. The 4

5 nonconvex program 5) may be relaxed further to a emidefinite program by replacing the nonconvex contraint rank X) = k with the linear contraint tr X) = k: max N N i= j= X ij.t. Xe e, X ij = 0, i, j) / E.t. i j tr X) = k, X 0. For every k-dijoint-clique ubgraph of G compoed of clique of ize r,..., r k, 6) define a feaible olution X R N N for 9) uch that { Xij /rl if both i, j belong to clique l = 0 otherwie. By contruction, the objective value of 9) correponding to X i equal to the number of node in the k-dijoint-clique ubgraph and rank X ) = k. Uing the Karuh-Kuhn- Tucker condition, we will derive condition for which X correponding to a k-dijoint-clique ubgraph of G, a defined by 6), i optimal for the convex relaxation of maximum node k- dijoint-clique problem given by 9). In particular, thee condition are ummarized by the following theorem. Theorem 2. Let X be feaible for 9). Suppoe alo that there exit λ R N +, µ R, η R N N and S Σ N + uch that ee T + λe T + eλ T + µi + η ij e i e T j = S, 0) i,j)/ E i j 9) λ T X e e) = 0, ) S, X = 0. 2) Here Σ N + i the cone of N N poitive emidefinite matrice,, i the trace inner product on R N N defined by Y, Z = tr Y Z T ) for all Y, Z R N N, e i denote the ith column of the identity matrix in R N N for all i =,..., N, and e i the all-one vector in R N. Then X i an optimal olution of 9). We omit the proof of thi theorem, a it i nothing more than the pecialization of the KKT condition [4] in convex programming to 9). 2. Contruction of the auxiliary matrix S Our proof technique to how that X i optimal for 6) i to contruct multiplier to atify Theorem 2.. The difficult multiplier to contruct i S, the dual emidefinite matrix. The reaon i that S mut imultaneouly atify homogeneou linear equation given by S, X = 0, requirement on it entrie given by the gradient equation 0), and poitive emidefinitene. 5

6 In thi ubection, we will lay the groundwork for our definition of S; in particular, we contruct an auxiliary matrix S. The actual multiplier ued to prove the optimality of X, a well a the proof itelf, are in the next ubection. Our trategy for atifying the requirement on S i a follow. The matrix S will be contructed in block with ize inherited from the block of X. In particular, let the node contained in the k planted clique be denoted C,..., C k, and let the remaining node be C k+. Then according to 6), X ha diagonal block XC q,c q for q =,..., k coniting of multiple of the all matrix. The remaining block of X are 0. The diagonal block of S will be perturbation of the identity matrix, with the rank-one perturbation choen o that each diagonal block of S, ay S Cq,Cq i orthogonal to X Cq,Cq. The entrie of an off-diagonal block, ay S Cq,C mut atify, firt of all, 0). Thi contraint, however, i binding only on the entrie correponding to edge in G, ince entrie correponding to abent edge are not contrained by 0) thank to the preence of the unbounded multiplier η ij on the left hand ide. Thee entrie that are free in 0) are choen o that 2) i atified. It i a well known reult in emidefinite programming that the requirement S, X = 0, X, S Σ N + together imply SX = XS = 0. Thu, the remaining entrie of S mut be choen o that X S = SX = 0. Becaue of the pecial form of X, thi i equivalent to requiring all row and column um of S Cq,C to equal zero. We parametrize the entrie of S Cq,C that are not predetermined by 0) uing the entrie of two vector y q, and z q,. Thee vector are choen to be the olution to ytem of linear equation, namely, thoe impoed by the requirement that X S = SX = 0. We how that the ytem of linear equation may be written a a perturbation of a linear ytem with a known olution, and we can thu get bound on y q, and z q,. The bound on y q, and z q, in turn tranlate to bound on S Cq,C, which are neceary to etablih the poitive emidefinitene of S. Thi emidefinitene i etablihed by proving that the diagonal block, which are identity matrice plu rank-one perturbation, dominate the off-diagonal block. Recalling our notation introduced earlier, G = V, E) ha a k-dijoint-clique ubgraph K compoed of clique C, C 2,..., C k of ize r, r 2,..., r k repectively. Let C k+ := V \ k i=c i ) be the et of node of G not in K and let r k+ := C k+. Let N := V. Let ˆr := min{r, r 2,..., r k }. For each v V, let n v denote the number of node adjacent to v in C for all {,..., k + }, and let clv) denote index i {,..., k + } uch that v C i. Let AḠ) RN N be the adjacency matrix of the complement Ḡ of G; that i [AḠ)] i,j = if i, j) / E and 0 otherwie. Next, fix q, {,..., k + } uch that q. Let H = H q, R Cq C be the block of AḠ) with entrie indexed by the vertex et C q and C, and let D = D q, R Cq Cq be the diagonal matrix uch that, for each i C q, the i, i)th entry of D i equal to the number of node in C not adjacent to i. That i D = r I Diag n C q ) where n C q R Cq i the vector with ith entry equal to n i for each i C q. Let F = F q, = D,q. 6

7 Next, define the calar c = c q, := ˆr 2 ˆr 2 r q + r ˆr + r q ), if k ), otherwie. Next, for each q, =,..., k + uch that q let b = b q, R Cq C be defined by [b q, ] i = c Note that the matrix D H H T F { n i, if i C q n q i, if i C. i weakly diagonally dominant ince the ith row of H contain exactly r n i, and, hence, poitive emidefinite. Further, let y = y q, and z = z q, be a olution of the perturbed ytem ) ) D + θee T H θee T y H T θee T F + θee T = b 3) z for ome calar θ > 0 to be defined later. The rationale for thi ytem of equation 3) i a follow. Below in 8), we hall define the matrix S Cq,C according to the formula that entrie i, j) correponding to edge in E are et to c q,, while entrie i, j) correponding to abent edge are et to the um [y q, ] i + [z q, ] j. Matrix S Cq,C ha the ame formula; refer to 25) below. A mentioned earlier, it i required that all row and column um of S Cq,C equal zero. Conider, e.g., the um of the entrie in a particular row i C q. Thi um conit of r term; of thee term, n i of them are c q, correponding to edge from i to C ) while the other r n i have the form [y q, ] i + [z q, ] j. Thu, the requirement that the row um to zero i written n i c q, + [y q, ] i + [z q, ] j ) = 0 which may be rewritten j C ;i,j)/ E r n i )[y q, ] i + ) j C ;i,j)/ E [z q, ] j = n i c q,. 4) Equation 4) i exactly a row of 3) in the cae θ = 0 becaue of the formula ued to define D, F, H, b. In the cae that θ i not zero, the equation for the ith row in 3) ha an additional term of the form θe T y q, e T z q, ). Thi additional term doe not affect the reult, a the following argument how. The verion of 3) with θ = 0 i ingular becaue the vector e; e) i in it null pace. Now fix θ > 0 uch that 3) i noningular. By the fact that e; e) i in the null pace of the coefficient matrix ) D H H T, F 7

8 taking the inner product of each ide of 3) with e; e) yield θr q + r )e T y e T z) = b T e b T 2 e, where b R Cq, b 2 R C are the vector of entrie of b correponding to C q and C repectively. Moreover, b T e b T 2 e = i C q n i j C n q j = 0 becaue the number of edge entering C q from C i equal to the number of edge entering C from C q. Therefore, if for ome θ > 0 we are able to how that 3) i noningular which we hall etablih in Section 3 and again in Section 4) then thi particular y q,, z q, ) atifying 3) will alo be a olution to 4) ince the additional term θe T y q, e T z q, ) i zero. For the remainder of thi ection, in order to formulate definition for the remaining multiplier, aume that θ > 0 and that 3) i noningular. Furthermore, aume that D ii > 0 for all i C q and F ii > 0 for all i C. Let A = Aθ) := D + θee T 0 0 F + θee T ), P = P θ) := then we have aumed that A + P i noningular, and ) y = A + P ) b. z 0 H θee T H T θee T 0 The proof technique in Section 3 and 4 i to how that Q := A+P ) A i mall o that y, z) i cloe to A b. Let Q = Q T, Q T 2 ) T where Q R Cq Cq C) and Q 2 R C Cq C). Then, under thi notation, y z ) = A b + = Q Q 2 ) b D + θee T ) 0 0 F + θee T ) Therefore, if D, F and A + P are noningular, and ) ) b + b 2 Q Q 2 ) b. y = D + θee T ) b + Q b = I + θd ee T ) D b + Q b z = I + θf ee T ) F b 2 + Q 2 b. Let ȳ := y Q b and z := z Q 2 b. In order to give explicit formula for ȳ and z we ue the well-known Sherman-Morrion-Woodbury formula ee, for example, [2, Equation 2..4]), tated in the following lemma, to calculate I + θd ee T ) and I + θf ee T ). Lemma 2. If A i a noningular matrix in R n n and u, v R n atify v T A u then A + uv T ) = A A uv T A + v T A u. 5) 8 ) ;

9 and A an immediate corollary of Lemma 2., notice that ) ) ȳ = D θd ee T D b + θe T D = D I θeet D b e + θe T D. 6) e z = F θf ) ) ee T F b + θe T F 2 = F I θeet F b e + θe T F 2. 7) e Finally, we define the k + ) k + ) block matrix S R N N a follow: σ ) For all q {,..., k}, let S Cq,C q = 0. σ 2 ) For all q, {,..., k} uch that q, let S Cq,C = H q, y q, e T + ez q, ) T ) + c q, H q, ee T ). 8) σ 3 ) For all q {,..., k} and i C q, j C k+, let [ S Cq,Ck+ ] ij = [ S c q,k+, Ck+,C q ] ji = c q,k+ n q j /r q n q j ), if i, j) E otherwie. σ 4 ) Finally, for all i, j C k+, chooe [ S Ck+,C k+ ] ij = {, if i, j) E or i = j γ, if i, j) / E 9) for ome calar γ to be defined later. We make a couple of remark about σ 2 ). A already noted earlier, thi formula define entrie of S Cq,C to be c q, in poition correponding to edge, and [y q, ] i + [z q, ] j in other poition. The vector y q, and z q, are defined by 3) preciely o that, when ued in thi manner to define S Cq,C, it row and column um are all 0 o that X S = SX = 0; the relationhip S Cq,C S Cq,C i given by 25) below). The ytem i quare becaue the number of contraint on S q, impoed by X S = SX = 0 after the predetermined entrie are filled in i C q + C one contraint for each row and column), which i the total number of entrie in y q, and z q,. A mentioned earlier, there i the light additional complexity that thee C q + C equation have a dependence of dimenion, which explain why we needed to regularize 3) with the addition of the θee T term. A a econd remark about σ 2, we note that S Cq,C = S C T,C q. Thi i a conequence of our contruction detailed above. In particular, y q, = z,q, H q, = H,q, T and D q, = F,q for all q, =,..., k uch that q. 9

10 2.2 Definition of the multiplier, optimality and uniquene We finally come to the main theorem of thi ection, which provide a ufficient condition for when the k-dijoint-clique ubgraph of G compoed of the clique C,..., C k i the maximum node k-dijoint-clique ubgraph of G. Theorem 2.2 Suppoe that G = V, E) ha a k-dijoint-clique ubgraph G compoed of the dijoint clique C,..., C k and let C k+ := V \ k i=c i ). Let r i = C i for all i =,..., k +, and let ˆr = min i=,...,k {r i }. Let X be the matrix of the form 6) correponding to the k- dijoint-clique ubgraph generated by C,..., C k. Moreover, uppoe that the matrix S a defined by σ ),..., σ 4 ) atifie S ˆr. 20) Then X i optimal for 9), and G i the maximum node k-dijoint-clique ubgraph of G. Moreover, if S < ˆr and n q v < r q 2) for all v V and q {,..., k} clv) then X i the unique optimal olution of 9) and G i the unique maximum node k-dijoint-clique ubgraph of G. Proof: We will prove that 20) i a ufficient condition for optimality of X by defining multiplier µ, λ, η, and S and proving that if 20) hold then thee multiplier atify the optimality condition given by Theorem 2.. Let u define the multiplier µ and λ by for all q =,..., k and µ = ˆr = min{r, r 2,..., r k }, 22) λ i = ˆr/r q) 2 for all i C k+. Notice that by our choice of µ and λ we have for all i C q, 23) λ i = 0. 24) S Cq,C q = ˆrI ˆr/r q )ee T for all q =,..., k by 0). Moreover, we chooe η uch that { Sij λ η ij = i λ j +, if i, j) / E, i j 0, otherwie for all i, j V. Note that, by our choice of η, we have { SCq,C S Cq,C =, if q, {,..., k + }, q S Ck+,C k+ + ˆrI, if q = = k +. 25) by 0). 0

11 By contruction, µ, λ, η, and S atify 0). Since the ith row um of X i equal to for all i C q for all q =,... k and i equal to 0 for all i C k+, X and λ atify the complementary lackne condition ). Moreover, X, S = k [S Cq,C r q ] i,j = q= q i Cq j C q k r q= q )) ˆr r qˆr rq 2 = 0, r q and thu X and S atify 2). It remain to prove that 20) implie that S i poitive emidefinite. To prove that S i poitive emidefinite we how that x T Sx 0 for all x R N if S atifie 20). Fix x R N and decompoe x a x = x + x 2 where { φi e, i {,..., k} x C i ) = 0, i = k + for φ R k choen o that x 2 C i ) T e = 0 for i =..., k, x 2 C k+ ) = xc k+ ). Here, xc i ), x C i ), and x 2 C i ) denote the vector in R C i compoed of the entrie indexed by C i of x, x, and x 2 repectively, for each i =,..., k +. Note that x i in the column pace of X. Then x i in the null pace of S and we have x T Sx = x T 2 Sx 2 = ˆr x x T 2 Sx 2 ˆr S ) x 2 2, ince x 2 C i ) i orthogonal to e for all i =,..., k. Therefore, S i poitive emidefinite, and, hence, X i optimal for 9) if S ˆr. Now uppoe that S < ˆr and, for all i =,..., k, no node in C i i adjacent to every node in ome other clique. Then X i optimal for 9). For all i =,..., k, let v i R N be the characteritic vector of C i. That i, {, if j Ci [v i ] j = 0, otherwie. Notice that X = k i= /r i)v i v i ) T. Moreover, by complementary lackne, X, S = 0 and, thu, v i i in the null pace of S for all i =,..., k. On the other hand, conider nonzero x R N uch that x T v i = 0 for all i =,..., k. That i, x i orthogonal to the pan of {v i : i =,..., k}. Then x T Sx = ˆr x 2 + x T Sx ˆr S ) x 2 > 0. Therefore, NullS) = pan{v i : i =,..., k} and rank S) = N k. Now uppoe that ˆX i alo optimal for 9). Then, by complementary lackne, ˆX, S = 0 which hold if and only if ˆXS = 0. Therefore, the row and column pace of ˆX lie in the null pace of S. It follow immediately, ince ˆX 0, that ˆX can be written in the form ˆX = k σ i v i vi T + i= k i= k ω i,j v i vj T j= j i

12 for ome σ R k + and ω Σ k, where Σ k denote the et of k k ymmetric matrice. Now, if ω i,j 0 for ome i j then every entry in the block ˆXC i, C j ) = ˆXC j, C i ) T mut be equal to ω i,j. Since each of thee entrie i nonzero, thi implie that each node in C i i adjacent to every node in C j, contradicting Aumption 2). Therefore, ˆX ha ingular value decompoition ˆX = σ v v T + + σ k v k vk T. Moreover, ince ˆX i optimal for 9) it mut have objective value equal to that of X and thu k r i = i= N N Xi,j = i= j= N N i= j= ˆX i,j = k σ i ri 2. 26) i= Further, ince ˆX i feaible for 9), σ i r i 27) for all i =,..., k. Combining 26) and 27) how that σ i = /r i for all i =,..., k and, hence, ˆX = X a required. 3 The Adverarial Cae Suppoe that the graph G = V, E) i generated a follow. We firt add k dijoint clique with vertex et C,..., C k of ize r, r 2,..., r k repectively. Then, an adverary i allowed to add a et C k+ of additional vertice and a number of the remaining potential edge to the graph. We will how that our adverary can add up to Oˆr 2 ) noie edge where ˆr = min{r,..., r k } and the k-dijoint-clique ubgraph compoed of C,..., C k will till be the unique maximum k-dijoint-clique ubgraph of G. The main theorem concerning the adverarial cae i a follow. Theorem 3. Conider an intance of the k-dijoint-clique problem contructed according to the preceding decription, namely, G contain a k-dijoint-clique graph G whoe node are partitioned a C C k where C q = r q, q =,..., k, plu additional node denoted C k+ and additional edge which may have endpoint choen from any of C,..., C k+ ). Let ˆr = minr,..., r k ). Aume the following condition are atified:. For all q =,..., k, i C q, for all {,..., k + } q, Here, δ i a calar atifying 29) below. n i δ minr q, r ). 28) 2. EG) \ EG ) ρˆr 2, where ρ i a poitive calar depending on δ. Then X given by 6) i the unique optimal olution to 9), and G i the unique optimal olution of the k-dijoint clique problem. 2

13 We remark that two of the condition impoed in thi theorem are, up to the contant factor, the bet poible according to the following information-theoretic argument. Firt, if n i = r, then node i could be inerted into clique, o the partitioning between C and C q would no longer be uniquely determined. Thi how the neceity of the condition n i Or ). The condition that EG)\EG ) ρˆr 2 i neceary, up to the contant factor, becaue if EG) \ EG ) ˆrˆr )/2, then we could interconnect an arbitrary et of ˆr node choen from among the exiting clique with edge to make a new clique out of them, again poiling the uniquene of the decompoition. An argument for the neceity of the condition that n i δr q i not apparent, o poibly there i a trengthening of thi theorem that drop that condition. The remainder of thi ection i the proof of thi theorem. A might be expected, the proof hinge on etablihing 20); once thi inequality i etablihed, then Theorem 2.2 complete the argument. A before, let r k+ denote C k+. For the remainder of the proof, to implify the notation, we aume that r k+ 2ρˆr 2. The reaon i that ince EG)\EG ) ρˆr 2 by aumption, if r k+ > 2ρˆr 2 then C k+ would include one or more iolated node i.e, node of degree 0), and thee node can imply be deleted in a preliminary phae of the algorithm. The algorithm till work with an arbitrary number of iolated node in G \ G, but the notation in the proof require ome needle additional complexity.) Recall that the contruction of the multiplier preented in Section 2 depended on two calar θ in 3) and γ in 9): chooe θ = and γ = 0. We impoe the aumption that δ 0, 0.382). The contant i choen o that 0 < δ < δ) 2. 29) We will how that, under thi aumption, there exit ome β > 0 depending only on δ uch that S Cq,C 2 β b q, for all q, {,..., k} uch that q. Chooe q, {,..., k} uch that q and let D, F, H, b, and c be defined a in Section 2.. Without lo of generality we may aume that r q r. Moreover, let y and z be the olution of the ytem 3) and define A, Q, P a in Section 2.. We begin by howing that, under thi aumption, y and z are uniquely determined. Note that, ince n i = r D ii δr for all i C q and n q i = r q F ii δr q for all i C by Aumption 28), D and F are noningular and hence A i noningular. Moreover, A + P = AI + A P ) and, hence, A + P i noningular if A P <. Note that, for all t > 0, we have λ min D + tee T ) λ min D) = min i C q D ii 30) ince ee T 0 where λ min D + tee T ) i the mallet eigenvalue of the ymmetric matrix D + tee T. Taking t = in 30) how that D + ee T ) D = 3 min i Cq D ii δ)r 3)

14 ince, for each i C q, we have by Aumption 28). Similarly, δ)r D ii r F + ee T ) F = Combining 3) and 32) we have min j C F ii δ)r q. 32) A = max{ D + ee T ), F + ee T ) } δ) min{r q, r } =. 33) δ)r q On the other hand, P = H ee T H ee T F = H ij ) 2 i Cq j C /2 δr q 34) ince H ij i equal to in the cae that i, j) E and 0 otherwie and there at mot δrq 2 edge between C q and C by Aumption 28). Therefore, ince δ < δ) 2 by Aumption 29), we have δ A P A P δ < and, thu, A + P i noningular and y and z are uniquely determined. Now, recall that S Cq,C = H ye T + ez T ) cee T H). In order to calculate an upper bound on S Cq,C we write S Cq,C a S Cq,C = m + m 2 + m 3 + m 4 + m 5 35) where m := H ȳe T ), m 2 := H e z T ), m 3 := H Q be T ), m 4 := H eq 2 b) T ), m 5 := cee T H) and apply the triangle inequality to obtain 36) S Cq,C 5 m i. 37) i= Throughout our analyi of S Cq,C we will ue the following erie of inequalitie. For any W R m n, u R m and v R n, we have W uv T = Diag u) W Diag v) Diag u) Diag v) W = u v W. 38) 4

15 On the other hand, and W uv T = Diag u) W Diag v) v Diag u) W v Diag u) W F m ) /2 = v u 2 i W i, :) 2 u v i= max i=,...,m 39) W i, :) 40) W uv T u v max W :, j) 4) j=,...,n where W i, :) and W :, j) denote the ith and jth row and column of W. We begin with m. Applying the bound 40) with W = H, u = ȳ, and v = e we have Here, we ued the fact that max i Cq Hi, :) = max i Cq D /2 ii exactly r n i. Thu, ince it follow immediately that ȳ D + ee T ) b m 2 max i C q D ii ȳ 2. 42) m 2 b min i Cq D ii b δ)r, ince D ii r for all i C q. By an identical calculation, we have m 2 2 Next, applying 40) with W = H, u = Q b, v = e yield ince the ith row of H contain δ) 2 r b 2 43) δ) 2 r q b ) m 3 2 max i C q D ii Q b 2 r Q b 2 r Q 2 b 2 ince max i D ii r. To derive an upper bound on m 3 2 we firt derive an upper bound on Q. Note that Q = A + P ) A = I + A P ) I)A = A P ) l A 45) ince I + X) = l=0 X)l for all X uch that X < by Taylor Theorem. Notice that ) 0 A P P = P l=

16 where P = D + θee T ) H θee T ), P 2 = F + θee T ) H T θee T ). It follow immediately that ) P P Q = 2 ) l+ 0 0 P 2 P ) l+ l=0 0 P P 2 ) l P P 2 P ) l P 2 0 )) A 46) ince, for any integer l 0 P P 2 0 ) l = P P 2 ) l/2 0 0 P 2 P ) l/2 0 P P 2 ) l )/2 P P 2 P ) l )/2 P 2 0 ), if l even ), if l odd. Therefore, and Q D + θee T ) P P 2 l + P F + θee T ) P P 2 l 47) l= Q 2 F + θee T ) P P 2 l + P 2 D + θee T ) P P 2 l. 48) l= Subtituting 3), 32) and 34) into 47) yield where ince Q δ)r δ ) l + δ /2 δ) 2 δ) 2 r l= c = 2 max{δ/ δ), δ} δ) 2 δ P P 2 H ee T 2 D F l=0 l=0 ) l δ c/r δ) 2 49) l=0 δ δ) 2. Note that Aumption 29) enure that the infinite erie in 49) converge. It follow that m 3 2 c2 r q b 2. 50) On the other hand, Q 2 δ)r q c/r q δ ) l + δ δ) 2 δ) 2 r l= δ δ) 2 l=0 ) l 5) 6

17 ince r r q. Thu, applying 4) with W = H, u = e, v = Q 2 b we have Finally, m 4 2 r q Q 2 2 b 2 c2 r q b 2. 52) m 5 2 = ch ee T ) 2 ch ee T ) 2 F = c H ij ) 2 i C q j C = c i C q n i = b. 53) Therefore, there exit β R uch that S Cq,C 2 β b 2 r q + b 54) where β depend only on δ. Moreover, ince b 2 b b and { } b = c max max n i, max n q i δc min{r q, r } = δcr q 55) i C q i C by Aumption 28), there exit β depending only on δ uch that S Cq,C 2 β b 56) a required. Next, conider S Cq,Ck+ for ome q {,..., k}. Recall that { [ S c, if i, j) E Cq,Ck+ ] ij = cn j /r q n j ), otherwie where n j = n q j i the number of edge from j C k+ to C q for each j C k+. Hence, S Cq,Ck+ 2 S Cq,Ck+ 2 F = ) ) 2 n j c 2 nj c + r q n j ) r q n j j C k+ c 2 n j + δn ) j δ) j C k+ = c2 δ EC q, C k+ ) 57) where EC q, C k+ ) i the et of edge from C q to C k+. Similarly, by our choice of γ = 0 in σ 4 ), we have S Ck+,C k+ 2 S Ck+,C k+ 2 F = r k+ + 2 EC k+, C k+ ). 58) 7

18 Let B be the vector obtained by concatenating b q, for all q, {,..., k}. Then, there exit calar ĉ, ĉ 2 R depending only on δ uch that k+ k+ S Cq,C 2 = q= = q, {,...,k} q S Cq,C k S Cq,Ck+ 2 + S Ck+,C k+ 2 q= k+ ĉ B + ĉ 2 EC q, C k+ ) + r k+ q= by 56), 57) and 58). It follow that, ince b q, 2 EC q, C ) for all q, {,..., k} uch that q, there exit ĉ 3 0 depending only on δ uch that k+ k+ S Cq,C 2 ĉ 3 R + r k+ q= = where R := EG)\EG ) i the number of edge of G not contained in the k-dijoint-clique ubgraph G compoed of C,..., C k. The hypothei of the theorem i that R ρˆr 2. We have alo aumed earlier that r k+ 2ρˆr 2. Hence the um of the quare of the 2-norm of the block of S i at mot ĉ 3 + 2)ρˆr 2. Therefore, there exit ome ρ > 0 depending only on δ uch that the preceding inequality implie S ˆr. Thi prove the theorem. 4 The Randomized Cae Let C, C 2,..., C k+ be dijoint vertex et of ize r,..., r k+ repectively, and let V = k+ i= C i. We contruct the edge et of the graph G = V, E) a follow: Ω ) For each q =,..., k, and each i C q, j C q uch that i j we add i, j) to E. Ω 2 ) Each of the remaining poible edge i added to E independently at random with probability p 0, ). Notice that, by our contruction of E, the graph G = V, E) ha a k-dijoint-clique ubgraph G with clique indexed by the vertex et C,..., C k. We wih to determine which random graph G generated according to Ω ) and Ω 2 ) have maximum node k-dijointclique ubgraph equal to G and can be found with high probability via olving 9). We begin by providing a few reult concerning random matrice with independently identically ditributed i.i.d.) entrie of mean Reult on norm of random matrice Conider the probability ditribution P for a random variable x defined a follow: { with probability p, x = p/ p) with probability p. 8

19 It i eay to check that the mean of x i 0 and the variance of x i σ 2 = p/ p). In thi ection we provide a few reult concerning random matrice with entrie independently identically ditributed i.i.d.) according to P. We firt recall a theorem of Geman [] which provide a bound on the larget ingular value of a random matrix with independently identically ditributed i.i.d.) entrie of mean 0. Theorem 4. Let A be a yn n matrix whoe entrie are choen according to P for fixed y R +. Then, with probability at leat c exp c 2 n c 3 ) where c > 0, c 2 > 0, and c 3 > 0 are contant depending on p and y, for ome contant c 4. A c 4 n Note that thi theorem i not tated exactly in thi form in [], but can be deduced by taking k = n q for a q atifying 2α + 4)q < in the equation on pp A imilar theorem due to Füredi and Komló [9] exit for ymmetric matrice A with entrie ditributed according to P. Theorem 4.2 For all integer i, j, j i n, let A ij be ditributed according to P. Define ymmetrically A ij = A ji for all i < j. Then the random ymmetric matrix A = [A ij ] atifie A 3σ n with probability at leat to exp cn /6 ) for ome c > 0 that depend on σ. A in Theorem 4., the theorem i not tated exactly in thi manner in [9]; the tated form of the theorem can be deduced by taking k = σ/k) /3 n /6 and v = σ n in the inequality P max λ > 2σ n + v) < n exp kv/2 n + v)) on p Next, we tate a verion of the well-known Chernoff bound which provide a bound on the tail ditribution of a um of independent Bernoulli trial ee [5, Theorem 4.4 and 4.5]). Theorem 4.3 Chernoff Bound) Let X,..., X k be a equence of k independent Bernoulli trial, each ucceeding with probability p o that EX i ) = p. Let S = k i= X i be the binomially ditributed variable decribing the total number of uccee. Then for δ > 0 It follow that for all a 0, p k), ) ) e δ pk P S > + δ)pk. 59) + δ) +δ) P S pk > a k) 2 exp a 2 /p). 60) The final theorem of thi ection i a follow ee [2, Theorem 4]). 9

20 Theorem 4.4 Let A be an n N matrix whoe entrie are choen according to P. Suppoe alo that log N n. Let à be defined a follow. For i, j) uch that A ij =, we define à ij =. For entrie i, j) uch that A ij = p/ p), we take Ãij = n j /n n j ), where n j i the number of in column j of A. Then there exit c > 0 and c 2 0, ) depending on p uch that P A à 2 F c N) 2/3) N Nc n 2. 6) 4.2 A bound on S in the randomized cae Suppoe that the random graph G = V, E) containing k-dijoint-clique ubgraph G compoed of clique C,..., C k i contructed according to Ω ) and Ω 2 ) with probability p. Let C k+, r,..., r k+, ˆr, and N be defined a in Section 2.. Further, let θ = p in 3) and γ = p/ p) in 9). We begin by tating the main theorem of the ection. Theorem 4.5 Suppoe that G = V, E) ha a k-dijoint-clique ubgraph G compoed of the clique C,..., C k and let C k+ := V \ k i=c i ). Let r i = C i for all i =,..., k + and uppoe that r q ˆr 3/2 for all q =, 2,..., k where ˆr = min i=,...,k {r i }. Then there exit ome β, β 2 > 0 depending only on p uch that k ) /2 k ) /2 S β r 2 + β 2 N 62) r = q= q with probability tending exponentially to a ˆr approache. Thi theorem i meant to be ued in conjunction with Theorem 2.2. In particular, if the right-hand ide of 62) i le than ˆr, then the planted graph G may be recovered. It i clear from 20) and the econd term on the right-hand ide of 62) that the ufficient condition for uniquene and optimality given by Theorem 2.2 cannot be atified unle N = Oˆr 2 ). We now give a few example of value for r,..., r k+ that fulfill 62).. Conider the cae k =, i.e., a ingle large clique. In thi cae, taking r = cont N /2 atifie 20) ince the firt term on the right i ON /4 ). 2. Suppoe k > and r =... = r k = cont N α. In thi cae, the firt parentheized factor on the right in 62) i Ok /2 N α ) while the econd i Ok /2 N α/2 ), and therefore the firt term i OkN α/2 ). For 20) to hold, we need thi term to be Oˆr) = ON α ), which i valid a long a k = ON α/2 ). We alo need α /2 a noted above to handle the econd term on the right. For example, for α = /2 the algorithm can find a many a ON /4 ) clique of thi ize. For α = 2/3, the algorithm can find a many a ON /3 ) clique of thi ize, which i the maximum poible ince the clique are dijoint and N i the number of node. 3. The clique may alo be of different ize. For example, if there i one large clique of ize ON 2/3 ) and N /6 maller clique of ize ON /2 ), then ˆr = ON /2 ), the firt parentheized factor in 62) i N 2/3 while the econd i N /6, o the entire firt factor i ON /2 ) = Oˆr). 20

21 We note that the reult for random noie in the k-dijoint-clique problem are much better than the reult for adverary-choen noie. In the cae of adverary-choen noie, the number of allowable noie edge i bounded above by a contant time the number of edge in the mallet clique. In the cae of random noie, the number of allowable noie edge i bounded above by a calar multiple of the number of potential edge in the complement of the planted k-dijoint-clique ubgraph. Thu, the number of allowable random noie edge can be a much a the quare of the number of edge in the mallet clique e.g., if there are N /4 clique each of ize N /2, then the mallet clique ha ON) edge veru ON 2 ) random noie edge). We do not know whether the bound in 20) i the bet poible. For intance, there i no obviou barrier preventing the algorithm from recovering a many a N /2 planted clique of ize N /2 in a random graph, but our analyi doe not carry through to thi cae. The remainder of thi ection i devoted to the proof of Theorem 4.5. We write S a S = S + S 2 + S 3 + S 4 + S T 4 where S i R N N, i =,..., 4 are k + ) by k + ) block matrice uch that { [ S SCq,C ] Cq,C =, if q, {,..., k}, q 0, otherwie R Cq,C, if q, {,..., k} [ S Ŝ 2 ] Cq,C = Cq,Ck+, if = k + Ŝ Ck+,C, if q = k + S Ck+,C k+, if q = = k + { [ S RCq,C 3 ] Cq,C =, if q, {,..., k} 0, otherwie { [ S SCq,C 4 ] Cq,C = ŜC k+ q,c k+, if = k +, q {,..., k} 0, otherwie where R R N N i a ymmetric random matrix with independently identically ditributed entrie uch that {, with probability p R ij = p/ p), with probability p and Ŝ RN N uch that {, if i, j) E Ŝ ij = p/ p), otherwie. Notice that, by Theorem 4.2, there exit ome κ, κ 2, κ 3 > 0 uch that P S 2 + S ) 3 κ N κ 2 exp κ 3 N /6 ). 63) Moreover, by Theorem 4.4, there exit κ 4 > 0 and κ 5, κ 6, 0, ) uch that P S ) 4 κ 4 N κ N 5 + Nκ N 6. 64) 2

22 Hence, there exit ome calar β 2 depending only on p uch that S S + β 2 N with probability tending exponentially to a ˆr. It remain to prove that k ) /2 k ) /2 S = O = r 2 r q= q with probability tending exponentially to a ˆr approache. To do o, conider two vertex et C q and C uch that q. Without lo of generality we may aume that r q r. Define H = H q,, D = D q,, F = D,q, b = b q,, c = c q,, y, z, A, and P a in Section 2.. The following theorem provide an upper bound on the pectral norm of S Cq,C for q, that hold with probability tending exponentially to a ˆr approache. Theorem 4.6 Suppoe that r q and r atify Then there exit B > 0 depending only on p uch that r q r r 3/2 q. 65) r [ S ] Cq,C = S Cq,C B 66) rq with probability tending exponentially to a ˆr approache. The remainder of thi ection i devoted to the proof of Theorem 4.6; Theorem 4.5 will then be etablihed a an immediate conequence. Recall that S Cq,C = H ye T + ez T ) cee T H). We begin by howing that A + P i noningular and, hence, y and z are uniquely determined. Let δ := p)/2p). Recall that n i = r D ii correpond to r independent Bernoulli trial each ucceeding with probability equal to p and, hence, ) P n e δ pr i + δ)pr ) = P r D ii + δ)pr ) 67) + δ) +δ) for each i C q by Theorem 4.3. Rearranging, we have that D ii θ δp)r with probability at leat ) e δ pr + δ) +δ) for each i C q. Similarly, ) P n q i + δ)pr e δ prq q) = P F ii θ δp)r q ) 68) + δ) +δ) 22

23 for all i C. Therefore, by the union bound, r D ii + δ)pr for all i C q and r q F ii + δ)pr q for all i C, and, hence, D, F are noningular with probability at leat ) e δ prq ) e δ pr r r + δ) +δ) q + δ) +δ) r q + r ) e δ + δ) +δ) ) pˆr. 69) Moreover, applying 30) how that D + θee T and F + θee T are noningular and D + θee T ) D, θ δp)r 70) F + θee T ) F, θ δp)r q 7) with probability at leat 69). It follow immediately that A i noningular and A = max{ D + θee T ), F + θee T ) } θ δp) min{r q, r } = 72) θ δp)r q with probability at leat 69). Recall that, in the cae that A i noningular, it uffice to prove that A P < to how that A + P i noningular. Moreover, recall that θ = p i choen to enure that the entrie of H θee T have expected value equal to 0. We can extend H θee T to an r r random matrix P with entrie i.i.d. with expected value equal to 0 by adding r r q row with entrie i.i.d. uch that each additional entry take value equal to θ with probability p and value equal to p with probability p. Therefore, by Theorem 4. P = H θee T P γ r 73) for ome γ > 0 depending only on p with probability at leat c exp c r c 3 2 ) where c i > 0 depend only on p. Combining 72), 73), 65) and applying the union bound how that with probability at leat r q + r ) r A P = γ < θ δp)r q e δ + δ) +δ) ) pˆr c exp c 2 r c 3 ) for ufficiently large r q. Therefore, A+P i noningular and y and z are uniquely determined with probability tending exponentially to a ˆr. For the remainder of the ection we aume that A + P i noningular. We define Q, Q, Q 2, ȳ and z a in Section 2.. To find an upper bound on S Cq,C, we decompoe S Cq,C a S Cq,C = M + M 2 23

24 where M := H ȳe T + e z T ) cee T H) and M 2 := H Q be T + eb T Q T 2 ) We firt obtain an upper bound on the norm of M. We define d R Cq to be the vector uch that the entry d i i the difference between the number of edge added between the node i and C and the expected number of uch edge for each i C q. That i, d = n C q E[n C q ] = n C q pr e. Similarly, we let f := n q C pr q e. Note that, by our choice of d and f, we have r I D = pr I + Diag d) and r q I F = pr q I + Diag f). Notice that for θ = p we have D = θr I Diag d). Expanding 6) we have ) ȳ = D θd ee T D b + θe T D e D = + θe T D e b + θb e T D e θee T D b ) D = + θe T D e b + θb e T eb T )D e) ince e T D b = b T D e. Subtituting b = cr e d), where d := diag D), we have and, hence, ince ȳ = = cd b e T eb T = ce d T de T ) + θe T D e r e d + θe d T de T )D e) cd + θe T D e r e d + θee T e θ de T D e) cd Let ȳ := ω e, ȳ 2 := υ e where and let = + θe T D e r e + θr q e) ce = cr ) + θr q )D D + Diag d)) e ce + θe T D e θr ) cr + θr q ) cr + θr q ) = c e + D d + θe T D e)θr + θe T D e)θr I = θr D + Diag d)). ω = cθr q + r ) θr + r q ) c υ = ȳ 3 := cθr q + r ) + θe T D e)θr c ω cθr q + r ) + θe T D e)θr D d. 24

25 Hence, ȳ = ȳ + ȳ 2 + ȳ 3. Similarly, z = z + z 2 + z 3 where z := ω 2 e, z 2 := υ 2 e where and ω 2 = cr q + θr ) θr + r q ) c υ 2 = z 3 := cr q + θr ) + θe T F e)θr q c ω 2 cr q + θr ) + θe T F e)θr q F f. Therefore, we can further decompoe M a M = M + M 2 + M 3 where M := H ȳ e T + e z T ) cee T H), M 2 := H ȳ 2 e T + e z T 2 ), M3 := H ȳ 3 e T + e z T 3 ). Notice that the matrix M ha entrie correponding to edge equal to c and remaining entrie equal to cp/ p) ince ω + ω 2 = c + θ)r q + r ) θr q + r ) 2c = cp θ. Therefore, each entry of the matrix M ha expected value equal to 0. Moreover, each entry of the random block matrix ˆM of the form [ ] M ˆM = R ha expected value equal to 0 if R ha identically independently ditributed entrie uch that R i,j = { c, with probability p cp/ p), with probability p. Therefore, there exit c, c 2, c 3, c 4 > 0 uch that M ˆM c 4 r 74) with probability at leat c exp c 2 r c 3 ) by Theorem 4.. Next, to obtain upper bound on M 2 and M 3 we will ue the following lemma. Lemma 4. There exit B > 0 depending only on p uch that θr D ii α D B rq ii i C q r α/2 and θr q F ii α F B r ii i C q for α =, 2 with probability at leat where v p = e δ / + δ) +δ) ) p and δ = min{p, p p}. r α/2 75) 76) r q + r )vˆr p 22/3)ˆr 77) 25

26 Proof: We firt prove 75). For each j C q, let n j := n j. The random number {n j : j C q } are independent, and each i the reult of r Bernoulli trial, each with probability of ucce equal to p. We define Ψ to be the event that at leat one n j i very far from it expected value. That i, Ψ i the event that there exit j C q uch that n j > tr, where t = min{ p, 2p}. Moreover, we define Ψ to be it complement, and let ψn j ) be the indicator function uch that ψn j ) = {, if nj tr 0, otherwie. Let B be a poitive calar depending on p to be determined later. Then P θr D ii α B r ) q = P n i pr α B r q D ii r α/2 i Cq r n i r α/2 i Cq P n i pr α B r q r n Ψ + P Ψ). 78) i r α/2 i Cq We will analyze the two term eparately. For the firt term we ue a technique of Berntein ee [3]). Let φ be the indicator function of the nonnegative real. Then, P n j pr α B r ) q r n Ψ j r α/2 j Cq = P n j pr α B r q 0 r n ψn j r α/2 j ) = j C q j Cq = P j Cq = E φ j Cq r α/2 n j pr α r n j r α/2 n j pr α r n j Br q 0 ψn j ) = j C q Br q ψnj ). j C q Let h be a poitive calar depending p to be determined later. Notice that for any h > 0 and all x R, φx) exphx). Thu, by the independence of the n j, P n i pr α B r ) q r n Ψ i r α/2 i Cq ) n j pr α E exp h j Cq = exp h j C q E = f f rq r α/2 r n j ) B n j pr α r α/2 r n j ) B )) ψnj ) j C q ) ψn j ) 26

27 where )) ) n j pr α f j = E exp h r α/2 r n j ) B ψn j ). We now analyze each f j individually. Fix j C q. Then tr )) n j pr α f j = exp h r n j ) B P n j = i) i=0 tr i=0 exp h r α/2 i pr α p)r α/2 B )) P n j = i) ince i tr and, hence, i pr. We now reorganize thi ummation by conidering i uch that i pr < r, then i uch that r i pr < 2 r and o on. Notice that, ince i tr 2pr, we need only to conider interval until i pr reache pr. Hence, f j p r k=0 p r k=0 p r 2 k=0 i: i pr [k r,k+) r ) i: i pr [k r,k+) r ) exp h exp k + ) α p B h i pr α p)r α/2 B )) P n j = i) )) k + ) α exp h p B P n j = i) )) exp k 2 /p) by 60). Overetimating the finite um with an infinite um, we have ) hk + ) α f j 2 exp hb) exp p k2 /p. k=0 Chooing h uch that h p)/8p) enure that hk + ) α p k2 /p k 2 /2p) for all r q, r and k. Hence, plitting off the k = 0 term, we have ) h f j 2 exp p hb + 2 exp hb) exp k 2 /2p)). 79) Since k= exp k2 /2p)) i dominated by a geometric erie, the ummation in 79) i a finite number depending on p. Therefore, once h i choen, it i poible to chooe B, depending only on p and h, ufficiently large o that each of the two term in 79) i at mot /3. Therefore, we can chooe h and B o that f j 2/3 for all j C q. It follow immediately that P θr D ii α B r q D ii r α/2 i Cq Ψ 2/3) rq 2/3)ˆr. 80) 27 k=

28 To obtain a bound on the econd term in 78), notice that the probability that n j > tr i at mot vp r vˆr p where v p = e δ / + δ) +δ) ) p by Theorem 4.3, where δ = t/p = min{p, p p}. Thu, applying the union bound how that P θr D ii α B r q 2/3)ˆr + r vˆr D ii r α/2 q p. i Cq By an identical argument θr q F P ii α F B r ii i C q r α/2 ) 2/3)ˆr + r vˆr p. Applying the union bound one lat time how that 75) and 76) hold imultaneouly with probability at leat r q + r )vˆr p 22/3)ˆr a required. A an immediate corollary of Lemma 4., we have the following bound on υ and υ 2. Corollary 4. There exit B > 0 depending only on p uch that rq 3/2 + r 3/2 υ + υ 2 B r q + r )r q r ) /2 with probability at leat r q + r )vˆr p 22/3)ˆr. Proof: We begin with υ. Notice that υ = cθr q + r ) ) θ + θe T D e)r r q + r = cθr q + r )r q θr e T D e) θr r q + r ) + θe T D e). Moreover, θr e T D e r q = θr D ii i C q i C q θr D ii = θr D ii D ii i C q i C q and, ince D ii r for all i C q, we have r + θe T D e) r + θr ) q = θr q + r. 8) r Therefore, etting α = in 75) how that υ θr 3/2 cθr q + r )Br q r q + r ) + θe T D e) cbθr q + r )r q θ r r q + r )θr q + r ) r q B r r q + r ) 82) 28