Alexander Barvinok August 2008

Transcription

1 WHAT DOES A RANDOM CONTINGENCY TABLE LOOK LIKE? Alexander Barvinok August 2008 Abstract. Let R = (r,..., r m and C = (c,..., c n be positive integer vectors such that r r m = c c n. We consider the set Σ(R, C of non-negative m n integer matrices (contingency tables with row sums R and column sums C as a finite probability space with the uniform measure. We prove that a random table D Σ(R, C is close with high probability to a particular matrix ( typical table Z defined as follows. We let g(x = (x + ln(x + x ln x for x 0 and let g(x = P ij g(x ij for a non-negative matrix X = (x ij. Then g(x is strictly concave and attains its maximum on the polytope of non-negative m n matrices X with row sums R and column sums C at a unique point, which we call the typical table Z.. Introduction and the main result (. Random contingency tables. Let R = (r,..., r m be a positive integer m-vector and let C = (c,..., c n be a positive integer n-vector such that m r i = i= n c j = N. j= A contingency table with margins (R, C is a non-negative integer matrix D = (d ij with row sums R and column sums C: n d ij = r i for i =,..., m, j= m d ij = c j for j =,..., n i= d ij 0 and d ij Z for all i, j. 99 Mathematics Subject Classification. 5A52, 05A6, 60C05, 5A36. Key words and phrases. contingency table, random matrix, transportation polytope. This research was partially supported by NSF Grant DMS and a United States - Israel BSF grant Typeset by AMS-TEX

2 Let Σ(R, C be the set of all contingency tables with margins (R, C. As is well known, Σ(R, C is non-empty and finite. Let us consider Σ(R, C as a finite probability space endowed with the uniform probability measure. In this paper we address the following question: Suppose that D Σ(R, C is chosen at random. What D is likely to look like? The problem is interesting in its own right, but the main motivation comes from statistics, see [Go63], [DE85], [DG95] and references therein. A contingency table D = (d ij may represent certain statistical data (for example, d ij may be the number of people in a certain sample having the i-th hair color and the j- th eye color. One can condition on the row and column sums and ask what is special about a particular table D Σ(R, C, considering all tables in Σ(R, C as equiprobable, see [DE85]. To answer this question we need to know what does a random table D Σ(R, C look like. A considerable effort was invested in finding an efficient (polynomial time algorithm to sample a random table D Σ(R, C, see [DG95], [D+97], [C+06]. Despite a number of successes, such an algorithm is still at large in many interesting situations. In this paper, we do not discuss how to sample a random table but describe instead what is it likely to look like. The answer turns out to defy some entropy-based probabilistic intuition, see Section.6. We prove that a random contingency table D is close in a certain sense to some particular non-negative m n matrix Z, which we call the typical table. (.2 The typical table. Let P(R, C be the set of all m n non-negative matrices X = (x ij with row sums R and column sums C: n x ij = r i for i =,..., m, j= x ij 0 for all i, j. m x ij = c j for j =,..., n and i= Geometrically, P(R, C is a convex polytope of dimension (m (n, known as the transportation polytope. Let g(x = (x + ln(x + x ln x for x 0 and let g(x = ij g(x ij for a non-negative matrix X = (x ij. One can easily check that g is strictly concave and hence achieves its unique maximum Z = (z ij on P(R, C. We call Z the typical table with margins (R, C. Since the objective function g is concave, Z can be computed efficiently, both in theory and in practice, by existing methods of convex optimization, cf. [NN94]. 2

3 The solution Z to the above optimization problem was first introduced in author s paper [Ba07]. It was given the name of typical table (perhaps, with not enough justification in [B+08]. In this paper, we show that Z indeed captures some typical features of a random table D Σ(R, C. We prove our main result assuming certain regularity ( smoothness of margins. (.3 Smooth margins. Let us fix a number 0 < δ <. First, we assume that the dimensions m and n of the matrix are of the same order: (.3. m δn and n δm Second, we assume that the row sums and column sums are of the same order: (.3.2 δn m r i N δm δn n c j N δn for i =,..., m and for j =,..., n Finally, we assume that the density of the table is separated from 0: (.3.3 N mn δ. We say that the margins (R, C are δ-smooth if conditions (.3. (.3.3 are satisfied. This is a modification of the definition from [B+08]. We note that δ-smooth margins are also δ -smooth for any 0 < δ < δ. As we remarked (see (.3.3, we are interested in tables with the density separated from 0. For the case of sparse tables where r i n and c j m, see [Ne69], [GM07] and references therein. (.4 Definitions and notation. Let us choose a non-empty subset of entries of a matrix: S (i, j : i m, j n. For an m n matrix A = (a ij we let σ S (A = (i,j S to be the sum of the entries from S. The cardinality of a finite set X is denoted by X. a ij Now we state our main result. 3

4 (.5 Theorem. Let us fix numbers 0 < δ < /2 and κ > 0. Then there exists a positive integer q = q(δ, κ such that the following holds: Suppose that (R, C are δ-smooth margins such that m + n q. Let S (i, j : i m, j n be a set such that δmn S ( δmn, let Z be the typical table with margins (R, C, and let ln(m + n ɛ = δ (m + n. /3 Then Pr D Σ(R, C : ( ɛσ S (Z σ S (D ( + ɛσ S (Z 2(mn κ(m+n. In other words, asymptotically, as far as a sum over a positive fraction of entries is concerned, a contingency table D sampled uniformly at random from the set of contingency tables with given margins is very likely to be close to the typical table Z. (.6 The independence table. In [Go63], I.J. Good observes that the independence table Y = (y ij, y ij = r i c j /N for all i, j maximizes the entropy H(X = ij x ij N ln N x ij on the set of all matrices X = (x ij in the transportation polytope P(R, C. He argues that the maximum entropy principle implies that in the absence of any other information, the null hypothesis regarding a table D Σ(R, C should be the one stating that D is close to the independence table Y. The intuition appears to be that the majority of tables from Σ(R, C are close to Y, so the rejection of the null hypothesis results in the maximum information about D. Theorem.5 implies that the entropy based intuition can be corrected here: a random table D Σ(R, C is close to the typical table Z maximizing function g(x and not to the independence table Y maximizing the entropy H(X over the polytope P(R, C of non-negative matrices with row sums R and column sums C. One can show that Y = Z if and only if all row sums r i are equal or all column sums c j are equal. In fact, particular entries of the matrices Z and Y may demonstrate very different behavior even for reasonably looking margins. Suppose, 4

5 for example, that m = n, that r = c = 3n and that r i = c i = n for i >. Hence N = 3n + n(n = n 2 + 2n and for the independence table we have y = 9n2 n 2 + 2n 9. On the other hand, for the typical table Z the entry z grows linearly in n. Indeed, the optimality condition for Z (the gradient of g at Z is orthogonal to the affine span of the transportation polytope implies that ( zij + ln = λ i + µ j for all i, j z ij and some λ,..., λ n, µ,..., µ n. By symmetry, we can choose λ = µ = α and λ i = µ i = β for i >. Moreover, we must have 0 < α < β. Since z 2 > z 2j = e 2β for all j > and r 2 = n, we should have Therefore, z j = Since r = 3n we must have β > ln 2 2. e α+β < e β = for j >. 2 z > 3n n 2 > 0.58n. Let us show that the independence table Y and the typical table Z may also produce different asymptotic behavior of sums σ S (Y and σ S (Z as m and n grow and S is a subset of entries consisting of a positive fraction of all entries as in Theorem.5. For that, let us fixed some margins R = (r,..., r m and C = (c,..., c n such that z y. For a positive integer k let us consider the cloned margins ( R k = kr,..., kr,..., kr m,..., kr m k times k times ( C k = kc,..., kc,..., kc n,..., kr n. k times k times In particular, tables D Σ(R k, C k are km kn matrices with the total sum of entries equal to k 2 N, where N = r r m = c c n. Let S = S k be the set of entries in the upper left k k corner of a matrix from Σ(R k, C k, let Y k 5 and

6 be the independence table of margins (R k, C k and let Z k be the typical table of margins (R k, C k. It is not hard to show that σ S (Z k = k 2 z and σ S (Y k = k 2 y, so the ratio between the two sums remains fixed (and not equal to as k grows. We conjecture that the independence table Y is indeed close with high probability to a random table D Σ(R, C, if instead of the uniform distribution in Σ(R, C, a table D = (d ij is sampled from the Fisher-Yates probability measure, where ( m Pr (D = (N! r i! n c j! i= j= ij, d ij! cf. [DG95]. Compared to the uniform distribution, the Fisher-Yates measure weights down tables with large entries. (.7 Possible ramifications. Let us fix a subset W (i, j : i =,..., m; j =,..., n. Let us consider the set Σ(R, C; W of m n non-negative integer matrices D = (d ij with row sums R, column sums C and such that d ij = 0 for (i, j / W. Assuming that Σ(R, C; W is non-empty, we can consider Σ(R, C; W as a finite probability space with the uniform measure and ask what does a random table D Σ(R, C; W look like. As above, we define the typical table Z as the unique maximum of g(x on the polytope of non-negative matrices X = (x ij with row sums R, column sums C and such that x ij = 0 for (i, j / W. One can prove a version of Theorem.5 in this more general context for subsets S W. However, it appears that one has to assume, additionally, that there are no too large or too small values among the entries z ij of the typical table Z = (z ij, cf. the example in Section.6. In our case, when W is the set of all pairs (i, j, Lemma 2.3 ensures that the entries z ij are not too small while Proposition 5. ensures that they are not too large. In [Ba08] another variation of the problem is considered: what if we require d ij 0, for all i, j. It turns out that a random D is close to a particular matrix maximizing the sum of entropies of the entries among all matrices with row sums R, column sums C and entries between 0 and. In the rest of the paper, we prove Theorem.5. In Section 2, we recall main results of [Ba07] connecting the typical table Z with an asymptotic estimate for the number Σ(R, C of tables. In Section 3, we prove the key estimate. In Sections 4 and 5, we prove various technical estimates. In Section 6, we prove Theorem.5 under the additional assumption that N is bounded by a polynomial in m and n. In Section 7, we complete the proof of Theorem.5. 6

7 2. Preliminaries: an asymptotic formula for the number of tables In [Ba07], the following result was proved. (2. Theorem. Let R = (r,..., r m and C = (c,..., c n be positive integer vectors such that r r m = c c n = N. Let us define a function ( m n F (x, y = x r i i y c j j x i= j= ij i y j Then F (x, y attains its minimum for x = (x,..., x m and y = (y,..., y n. ρ(r, C = min 0<x,...,x m < 0<y,...,y n < F (x, y on the open cube 0 < x i, y j < and for the number Σ(R, C of non-negative integer m n matrices with row sums R and column sums C we have where γ > 0 is an absolute constant. ρ(r, C Σ(R, C N γ(m+n ρ(r, C, As is remarked in [Ba07], the substitution x i = e s i, y j = e t j transforms ln F (x, y into a convex function G(s, t = m r i s i + i= n j= c j t j ij ln ( e s i t j for s = (s,..., s m and t = (t,..., t n on the positive orthant R m + R n +. It turns out that the typical table Z is the solution to the problem that is convex dual to the problem of minimizing G. (2.2 Lemma. Let P = P(R, C be the polytope of m n non-negative matrices X = (x ij with row sums R and column sums C and let Z P(R, C be the typical table, see Section.2. Then one can write Z = (z ij, z ij = ξ iη j ξ i η j for all i, j and some 0 < ξ,..., ξ m ; η,..., η n < such that the minimum ρ(r, C of function F (x, y in Theorem 2. is attained at x = (ξ,..., ξ m and y = (η,..., η n : F (x, y = ρ(r, C = 7 min 0<x,...,x m < 0<y,...,y n < F (x, y.

8 Moreover, ρ(r, C = exp g(z. Proof. First, we show that matrix Z is strictly positive, z ij > 0 for all i, j. Indeed, suppose that z ij = 0 for some i and j. Let Y = (y ij, Y P, be a positive matrix, for example, y ij = r i c j /N. Since the derivative ( x + g (x = ln x is finite for x > 0 and equals + for x = 0 (we consider the right derivative here, we have g ( ( ɛz + ɛy > g(z for all sufficiently small ɛ > 0, which is a contradiction. Since Z lies in the relative interior of P, the gradient of g at Z must be orthogonal to the subspace of m n matrices with row and column sums equal to 0. Therefore, ( zij + ln = λ i + µ j for all i, j z ij and some λ,..., λ m and µ,..., µ n. Since λ i + µ j > 0 for all i, j, without loss of generality we assume that λ i, µ j > 0 for all i, j. Hence and since Z P, we have z ij = e λ i e µ j e λ i e µ j for all i, j (2.2. n j= m i= e λ i e µ j e λ i e µ j = r i for i =,..., m and e λ i e µ j e λ i e µ j = c j for j =,..., n. Equations (2.2. imply that s = (λ,..., λ m and t = (µ,..., µ n is a critical point of m n G(s, t = r i s i + ln ( e s i t j i= j= c j t j ij and since G is convex, (s, t is a minimum point of G on the positive orthant R m + R n +. Therefore, the point x = (ξ,..., ξ m and y = (η,..., η n, where ξ i = e λ i for i =,..., m and η j = e µ j for j =,..., n 8

9 is a minimum point of F (x, y on the open cube 0 < x i, y j <. It remains to notice that z ij = ξ iη j for all i, j ξ i η j and that g (z ij = (z ij + ln (z ij + z ij ln z ij ij ij ij = e λ i µ j ln ( e λ i µ j ij + e λ i e µ j ( e λ i µ j λ i + µ j + ln ( e λ i µ j ij m n = r i λ i + ln ( e λ i µ j i= =G (s, t. j= c j µ j ij We will need a lower bound for the entries of the typical table Z = (z ij proved in [B+08]. (2.3 Lemma. Let r + = max i, i=,...,m r = min i i=,...,m c + = max j, j=,...,n c = min j. j=,...,n Let Z = (z ij be the typical table. Then z ij r c r + m and z ij c r c + n 3. The key estimate and for all i, j. Recall that for an m n matrix A = (a ij and a subset S (i, j : i m, j n we let σ S (A = a ij. For a positive integer p, let h p (x,..., x n = (i,j S α,...,α n 0 α +...+α n =p x α xα n n be the complete symmetric polynomial of degree p in n variables x,..., x n. The main result of this section is the following estimate. 9

10 (3. Proposition. Let R = (r,..., r m and C = (c,..., c n be positive integer vectors such that r r m = c c n = N, let Z = (z ij be the typical table, and let S (i, j : be a set. Then for a positive integer p we have Σ(R, C D Σ(R,C N γ(m+n p!h p (z ij : (i, j S i =,..., m, j =,..., n σ S (D (σ S (D (σ S (D p + where h p is the complete symmetric polynomial of degree p in S variables and γ is the absolute constant from Theorem 2.., Let Ψ(R, C, S; τ = τ σs(d, D Σ(R,C where τ 0 is a real parameter. Our first observation is a standard exercise in generating functions. (3.2 Lemma. We have (i,j S τx i y j (i,j / S = Ψ(R, C, S; τx R y C, x i y j R,C where the sum is taken over all non-negative integer vectors R = (r,..., r m and C = (c,..., c n, x R = x r xr m and y C = y c yc n n and the series converges absolutely and uniformly on compact subsets of the set x i, y j < ɛ and τ +ɛ for any ɛ > 0. Next, we are going to differentiate the generating function of Lemma 3.2 in τ. (3.3 Lemma. Let s and p be positive integers and let 0 a,..., a s < be numbers. Let b k = a k for k =,..., s. a k Then ( s d p dτ p k= τa k τ= = p! ( s k= a k h p (b,..., b s, where h p is the complete symmetric polynomial of degree p in s variables. 0

11 Proof. We have ( s d p dτ p k= τa k = 0 m,...,m s p m +...+m s =p = 0 m,...,m s p m +...+m s =p = p! ( s k= p! m! m s! p! τa k s k= s k= d m k dτ m k ( a m k k ( τa k m k+ 0 m,...,m s p m +...+m s =p s k= τa k a m k k ( τa k m k and the proof follows. Now we are ready to prove Proposition 3.. (3.4 Proof of Proposition 3.. Let x = (ξ,..., ξ m and y = (η,..., η n be the point from Lemma 2.2, where the minimum of the function F (x, y = ( m i= x r i i n j= y c j j ij x i y j in Theorem 2. is attained. Hence we have 0 < ξ,..., ξ m ; η,..., η n <, z ij = ξ iη j ξ i η j for all i, j, and (3.4. Σ(R, C N γ(m+n ( m i= ξ r i i n j= η c j j ij. ξ i η j Differentiating the generating function in Lemma 3.2, we get (i,j / S ξ i η j dp dτ p (i,j S = ( d p ( dτ p Ψ(R, C, S; τ τ= m R,C i= τ= τξ i η j ξ r i i n j= η c j j.

12 Applying Lemma 3.3, we get (i,j / S ξ i η j dp dτ p (i,j S = p!h p (z ij : (i, j S ij τ= τξ i η j. ξ i η j Next we note that d p dτ p Ψ(R, C, S; τ τ= = D Σ(R,C since σ S (D is integer and hence σ S (D (σ S (D (σ S (D p + D Σ(R,C p!h p (z ( m ij : (i, j S The proof follows by (3.4.. σ S (D (σ S (D (σ S (D p + 0, i= ξ r i i n j= η c j j ij. ξ i η j 4. Bounding the complete symmetric polynomial In this section, we show that the value p!h p (a,..., a s of the complete symmetric polynomial in non-negative variables can be bounded by a moderate multiple of (a a s p assuming that s is sufficiently large compared to p and that none of the numbers a,..., a p is too big compared to the average. We prove the following main result. (4. Proposition. Let a,..., a s be numbers such that 0 a i α s a i for i =,..., s s and some α. Let p be an integer and suppose that s 8αp. Then ( s p p!h p (a,..., a s β(s, p a i where i= i= 4παp 2 α(p 2 β(s, p = + exp + p exp p. s s We recall that a random variable ξ has the standard exponential distribution if Pr ξ > t e t if t 0 = if t < 0. We start with a lemma. 2

13 (4.2 Lemma. Let ξ,..., ξ s be independent standard exponential random variables. Then for all a,..., a s we have ( s p p!h p (a,..., a s = E a i ξ i. Proof. The proof follows from the multinomial expansion and the observation that E ξi b = b! for a non-negative integer b. Next, we prove a concentration inequality for a linear function in exponential variables, cf. also Section 4.5 of [Le0]. (4.3 Lemma. Let ξ,..., ξ s be independent standard exponential random variables. Let a,..., a s be numbers such that a a s = and let i= 0 a i α s for i =,..., s and some α. Then s Pr a i ξ i + δ exp sδ2 i= s Pr a i ξ i + δ exp sδ i= for 0 δ and for δ. Proof. We use the Laplace transform method. Let us choose 0 < t s/2α. Then s s Pr a i ξ i + δ =Pr exp t a i ξ i exp t( + δ i= Here we used that i= exp t( + δe exp = exp t( + δ = exp exp s i= t( + δ tδ + t 2 s i= t s a i ξ i i= ta i s ln( ta i i= a 2 i exp ln( x x x 2 for 0 x /2. 3 tδ + αt2. s

14 Now, if δ we can choose t = sδ 2α s 2α and obtain If δ > we choose s Pr a i ξ i + δ exp sδ2. i= t = s 2α and obtain s Pr a i ξ i + δ i= exp sδ 2α + s exp sδ. Now we are ready to prove Proposition 4.. (4.4 Proof of Proposition 4.. Without loss of generality, we assume that a a s =. Let ξ,..., ξ s be independent standard exponential random variables. For t 0 we define s F (t = Pr a i ξ i t. i= Using Lemmas 4.2 and 4.3, we may write p!h p (a,..., a s = + p + p p + t p df (t = p 0 t p F (t dt t p exp t p exp t p F (t dt s(t 2 dt s(t dt.

15 We estimate the integrals 2 t p s(t 2 exp dt = = (t + p exp exp st2 exp st2 st2 dt + (p ln( + t + (p t exp st2 α(p 2 4πα = exp s s dt + (p t dt dt and + 2 t p s(t exp dt = = exp (t + p exp st exp st dt + (p ln( + t exp st + (p t p s. dt dt Summarizing, p!h p (a,..., a s + 4παp 2 s α(p 2 exp s + p exp p s. Since s 8αp, the proof follows. 5. Bounding the entries of the typical table In this section, we prove that large entries of the typical table Z belong to a small number of rows. (5. Proposition. Let R = (r,..., r m and C = (c,..., c n be positive vectors such that r r m = c c n = N and let Z = (z ij be the typical table. Let κ 2mn/N be a number. Suppose that c j N δn for j =,..., n 5

16 and some 0 < δ <. Let Then I = i : z ij κ N mn for some j. I 4m δκ. Proof. As in the proof of Lemma 2.2, we can write ( zij + ln = λ i + µ j for all i, j and some positive λ,..., λ m and µ,..., µ n. Let I 0 = i : λ i mn and J 0 = j : κn z ij z ij If i / I 0 then for all j we have ( zij + ln and hence z ij z ij < κ N mn. λ i > mn κn µ j mn. κn Therefore, I I 0. Similarly, if z ij κn/mn then j J 0. Hence without loss of generality, we may assume that J 0. Let us choose a j J 0. Then for any i I 0 we have ( zij + ln z ij 2mn κn. Hence for all i I 0 we have 2mn exp 4mn z ij κn κn (we used that e x + 2x for 0 x. Hence Since we conclude that z ij κn 4mn for i I 0 and j J 0. m z ij = c j N δn, i= I I 0 4c jmn κn 6 4m δκ.

17 6. Proof of Theorem.5 assuming that N is polynomially bounded In this section we prove Theorem.5 under one additional assumption that N is bounded by a polynomial in m and n, N (mn /δ. We start with a standard technical estimate. (6. Lemma. For a positive a > 0 and a positive integer p a/2 +, we have a(a (a p + a p p(p exp. a Proof. We have a(a (a p + = a p p k= ( k. a Moreover, p k= ( k p = exp a exp k= ( ln k a p(p a, exp 2 p k= k a where we use that ln( x 2x for 0 x /2. (6.2 Proof of Theorem.5 assuming that N (mn /δ. By Lemma 2.3, z ij δ 3 N mn for all i, j. Therefore, σ S (Z δ 4 N as long as S δmn. Our first observation is that it suffices to prove that under the conditions of the theorem, for any 0 < δ < /2 and for ln(m + n ɛ = δ (m + n /3 we have (6.2. Pr D Σ(R, C : σ S (D ( + ɛσ S (Z (mn κ(m+n provided m + n > q(δ, κ for some sufficiently large q(δ, κ. 7

18 Indeed, denoting the complement of S, we have S = (i, j : i m, j n \ S σ S (D = N σ S (D and σ S (Z = N σ S (Z and Pr D : σ S (D ( ɛσ S (Z =Pr D : σ S (D N ( ɛσ S (Z =Pr D : σ S (D σ S (Z + ɛσ S (Z. Since σ S (Z δ 4 N and σ S (Z N, we have that σ S (Z + ɛσ S (Z ( + ɛδ 4 σ S (Z. Therefore, Pr D : σ S (D ( ɛσ S (Z Pr D : σ S (D ( + ɛδ 4 σ S (Z and the bound Pr D Σ(R, C : σ S (D ( ɛσ S (Z (mn κ(m+n follows from (6.2. applied to S instead of S and δ 5 instead of δ. Our second observation is that it suffices to prove (6.2., assuming additionally that (6.2.2 z ij m /3 N mn for all (i, j S. Indeed, denoting S = (i, j S : z ij m /3 N mn we conclude by Proposition 5. that the number I of rows containing entries of S \ S satisfies I 4δ m 2/3 and hence σ S\ S(X 4δ 2 m /3 N for all non-negative matrices X with row sums R. Since σ S (Z δ 4 N we have σ S(Z ( 4δ 6 m /3 σ S (Z 8

19 In addition, S ( δmn 4δ 2 m /3. Since n δ m, the estimate (6.2. for S follows from that for S with a smaller δ > 0 and larger m and n. Finally, we prove (6.2. assuming ( Let Υ = Hence for all D Υ we have D (R, C : σ S (D ( + ɛσ S (Z. σ S (D σ S (Z δ 4 N δ 5 mn. Let us choose a positive integer p (to be adjusted later. By Proposition 3., Σ(R, C D Υ k= Let us choose p so that p (σ S (D k + Σ(R, C (6.2.3 p δ 5 mn/2. Then, by Lemma 6. for all D Υ we have p k= (σ S (D k + σ p S (D exp Next, we use Proposition 4. to bound D Σ(R,C k= p (σ S (D k + N γ(m+n p!h p (z ij : (i, j S p(p δ 5 mn ( + ɛ p σ p S (Z exp p(p δ 5. mn p!h p (z ij : (i, j S β(s, pσ p S (Z, where 4παp 2 α(p 2 β(s, p = + exp + p exp p, s s and we use s = S δmn, α = δ 3 m /3, and (6.2.4 p δ 4 m 2/3 n. 9.

20 Note that we can choose α = δ 3 m /3 since δ 3 N mn z ij m /3 N mn for all (i, j S. Therefore, (6.2.5 Using that we can choose Pr D Σ(R, C : = Υ Σ(R, C σ S (D ( + ɛσ S (Z ( + ɛ p N γ(m+n exp p(p δ 5 β(s, p. mn N (mn /δ ln(m + n and ɛ = δ (m + n /3 p C(δ, κ(m + n 4/3 for a sufficiently large constant C(δ, κ > 0 so that for a sufficiently large q = q(δ, κ and m + n q inequalities (6.2.3 and (6.2.4 hold and (6.2.5 implies ( Proof of Theorem.5 It remains to prove Theorem.5 in the case of a large (superpolynomial in m+n total sum N of entries. The idea of the proof is as follows: given margins (R, C with the total sum N of entries, we construct new margins (R, C with the sum of entries N bounded by a polynomial in m + n and a scaling map T : Σ(R, C Σ(R, C, which, roughly, scales every table D Σ(R, C by the same factor t. deduce Theorem.5 for margins (R, C from that for margins (R, C. We have R t R, C t C and T (D t D, where stands for rounding in some consistent way. In constructing map T we essentially follow the ideas of [D+97]. We then (7. Lattices, bases, and fundamental parallelepipeds. Let V be a finitedimensional real vector space and let Λ V be a lattice, that is, a discrete additive subgroup of V which spans V. Suppose that dim V = k and let u,..., u k be a basis of Λ. The set k Π = λ i u i : 0 λ i < for i =,..., k i= 20

21 is called the fundamental parallelepiped associated with the basis u,..., u k. Suppose that A is an affine space, dim A = dim V, on which V acts by translations: a + v A for all a A and v V and a + (v + v 2 = (a + v + v 2 for all a A and v, v 2 V. Let us choose a A. The set Λ a = a + Λ is called a point lattice in A. As is known, the translations v + Π : v Λ a cover A without overlapping. We will also use the following standard fact. Suppose that Λ Λ is a finer lattice and let Λ /Λ < be its index. Then, for any a, b A we have (a + Π (b + Λ = Λ /Λ, see for example Chapter VII of [Ba02]. Let us fix a point lattice Λ a A and a fundamental parallelepiped Π V of Λ. Given a point x A, we define its rounding y = x Λa,Π as the unique point y Λ a such that x y + Π. In our case, V is the space of real m n matrices with the row and column sums equal to 0, so dim V = (m (n, while A is the affine space of m n matrices with prescribed integer row and column sums, so that for all D A and U V we have D + U A. Furthermore, let Λ V be the lattice of integer matrices and let Λ A be the point lattice consisting of integer matrices. As is shown, for example, in [D+97], lattice Λ has a basis consisting of the matrices U ij for i n, j m that have in the (i, j and (i+, j+ positions, in the (i +, j and (i, j + positions and zeros elsewhere. Let Π be the fundamental parallelepiped of this basis U ij. We call this parallelepiped Π standard. We note that (7.. 2 x ij 2 for all i, j and all X Π, X = (x ij. Finally, for positive integer t let Λ = t Λ. Hence Λ /Λ = t (m (n. (7.2 The t-scaling map T. Let us choose a positive integer t and an arbitrary D 0 Σ(R, C, where R = (r,..., r m and C = (c,..., c n. Let us define a positive m n matrix B as follows. First, we obtain D by rounding up to the nearest integer every entry of t D 0 and adding 2 to the result. In particular, D is a positive integer matrix. Let B = D t D 0, so D = B + t D 0. Clearly, B = (b ij is an m n matrix with ( b ij < 3 for all i, j. Let R = (r,..., r m and C = (c,..., c n be the row and column sums of D respectively. Thus R and C are positive integer vectors and (7.2.2 t r i + 2n r i t r i + 3n for i =,..., m and t c j + 2m c j t c j + 3m for j =,..., n. 2

22 Let A be the affine subspace of matrices with row sums R and column sums C and let Λ A be the point lattice of integer matrices. Thus Λ = D + Λ, where Λ is the lattice of m n integer matrices with zero row and column sums, see Section 7.. For a matrix D Σ(R, C we define a matrix T (D by T (D = t D + B Λ,Π, where Π is the standard parallelepiped of Λ, see Section 7.. In words: given a table D Σ(R, C, matrix T (D is the unique integer matrix such that the translation T (D + Π of the standard parallelepiped Π contains t D + B. Clearly, T (D is an m n integer matrix with row sums R and column sums C. Moreover, since every entry of t D +B is at least 2 and because of (7.., matrix T (D is non-negative. Hence we defined a map T : Σ(R, C Σ(R, C. we summarize some of its properties below. (7.3 Lemma. ( For all Y Σ(R, C we have T (Y t (m (n ; (2 Let S (i, j : i =,..., m, j =,..., n be a set of indices. Then for all D Σ(R, C. t σ S (D σ S (T (D t σ S (D + 5 S Proof. Given Y Σ(R, C, we compute T (Y as follows: we consider the translation (Y B + Π of the standard parallelepiped Π and let T (Y = D : t D (Y B + Π and D is a non-negative integer matrix. Recall that Λ V is the lattice of m n integer matrices with the row and column sums equal to 0 and that Λ = t Λ. In the affine space of m n matrices with row sums t R and column sums t C let us consider the point lattice Λ = t D 0 +Λ consisting of matrices t D where D is an integer matrix. Then ((Y B + Π Λ = Λ /Λ = t (m (n and Part ( follows. Part (2 follows because of (7.. and (

23 (7.4 Lemma. Suppose that r i, c j (mn 2 for all i, j. Then, for any ζ 0 we have Pr D Σ(R, C : σ S (D tζ βpr Y Σ(R, C : σ S (Y ζ, where β > 0 is an absolute constant. Proof. By Part (2 of Lemma 7.3, if σ S (D tζ then σ S (Y ζ for Y = T (D. Using Part ( of Lemma 7.3, we can write D Σ(R, C : σs (D tζ Pr D Σ(R, C : σ S (D tζ = Σ(R, C Y Σ(R t (m (n, C : σ S (Y ζ Σ(R, C = Σ(R, C Σ(R, C t(m (n Pr Y Σ(R, C : σ S (Y ζ It is shown in [D+97] that for sufficiently large margins, the number of contingency tables is approximated within a constant factor by the volume of the corresponding transportation polytope, see Section.2. In particular, estimates of [D+97] imply that Σ(R, C β vol P(R, C and Σ(R, C β 2 vol P(R, C. for some absolute constants β, β 2 > 0. From (7.2.2, we have ( r i t(r i 3n tr i 3 m 2 n ( c j t(r i 3m tc j 3 mn 2 It follows then that for i =,..., m and for j =,..., n. vol P(R, C β 3 t (m (n vol P(R, C for some absolute constant β 3 > 0. The proof now follows. Next, we show that the t-scaling map T almost scales the typical table provided the margins R, C are large enough, that is, Z t Z. The idea of the proof is roughly the following: if margins (R, C and (R, C are large enough, then the corresponding typical tables Z and Z roughly optimize the functional ij ln x ij on the corresponding transportation polytopes and hence the map X tx roughly maps Z to Z. 23

24 (7.5 Lemma. Let Z = (z ij be the typical table of margins (R, C, let Z = ( z ij be the typical table of margins (R, C obtained by t-scaling and suppose that z ij (mn for all i, j. Then z ij tz ij β mn for all i, j and some absolute constant β > 0. Proof. First, we prove some useful inequalities for the function g(x = (x + ln(x + x ln x. We have g(tx g(x = tx x g (y dy = tx x ( y + ln y dy tx x dy y = ln(tx ln x = ln t. Also, g(x =(x + ln(x + (x + ln x + (x + ln x x ln x ( ( x + =(x + ln + ln x = ln x + + O for x. x x Finally, we note that Since from (7.2.2 we have g (x = x(x +. r i tr i and c j tc j for all i, j we have (7.5. max g(x ln t + max g(x. X P(R,C X P(R,C Let B be the matrix constructed in Section 7.2 and let W = t(z B P(R, C. Hence w ij t(mn 4 for all i, j. Since g (w ij = + ln w ij + O ( m 4 n 4 and g ( ( z ij = + ln z ij + O 24 m 4 n 4,

25 we have From (7.5. it follows that ( g(w = g(z + ln t + O m 3 n 3. ( (7.5.2 g(z g(w = O m 3 n 3. Next, we are going to exploit strong concavity of g and use the following standard inequality: if g (x α for some α > 0 and all a x b then ( a + b g α(b a2 g(a g(b. 8 If for some i, j we have w ij z ij (mn w ij, then in view of (7.5.2, for some point U on the interval connecting W and Z and all sufficiently large mn, we will have g(u > g(z, which is a contradiction. Thus z i w ij mn for all i, j and all sufficiently large mn. Since w ij tz ij 3 z ij 3 (mn 4, the proof follows. (7.6 Proof of Theorem.5. Without loss of generality we assume that N (mn 7 since the case of a polynomially bounded N is handled in Section 6. As in Section 6.2, it suffices to prove that under the conditions of the theorem we have (7.6. Pr D Σ(R, C : as long as m + n > q(δ, κ. Let us choose σ S (D ( + ɛσ S (Z (mn κ(m+n N t = (mn 6 25

26 and consider the t-scaling map T : Σ(R, C Σ(R, C. Since margins (R, C are δ-smooth, we have (mn 6 N (mn 7 and r i, c j (mn 4 for all i, j and all sufficiently large m + n. Let us choose 0 < δ < δ. It follows by (7.2.2 that the margins (R, C are δ -smooth for all sufficiently large m + n. Let Z be the typical table of (R, C, Z = ( z ij. As in Section 6.2, we have z ij (δ 3 N mn. Therefore, for all sufficiently large m + n we have z ij (mn Inequality (7.6. now follows by Lemmas 7.4, 7.5, and Theorem.5 applied to (R, C. Acknowledgment The author is grateful to Alexander Yong for his question in what sense the typical table introduced in [Ba07] and named so in [B+08] was typical. This paper is an attempt to answer that question. References [Ba02] [Ba07] [Ba08] [B+08] [C+06] [DE85] [DG95] [D+97] [Go63] A. Barvinok, A Course in Convexity, Graduate Studies in Mathematics, 54, American Mathematical Society, Providence, RI, A. Barvinok, Asymptotic estimates for the number of contingency tables, integer flows, and volumes of transportation polytopes, preprint arxiv: (2007. A. Barvinok, On the number of matrices and a random matrix with prescribed row and column sums and 0- entries, preprint arxiv: (2008. A. Barvinok, Z. Luria, A. Samorodnitsky, and A. Yong, An approximation algorithm for counting contingency tables, preprint arxiv: (2008. M. Cryan, M. Dyer, Martin, L.A. Goldberg, M. Jerrum, and M. Russell, Rapidly mixing Markov chains for sampling contingency tables with a constant number of rows, SIAM J. Comput. 36 (2006, P. Diaconis and B. Efron, Testing for independence in a two-way table: new interpretations of the chi-square statistic. With discussions and with a reply by the authors, Ann. Statist. 3 (985, P. Diaconis and A. Gangolli, Rectangular arrays with fixed margins, Discrete probability and algorithms (Minneapolis, MN, 993, IMA Vol. Math. Appl., 72, Springer, New York, 995, pp M. Dyer, R. Kannan, and J. Mount, Sampling contingency tables, Random Structures Algorithms 0 (997, I. J. Good, Maximum entropy for hypothesis formulation, especially for multidimensional contingency tables, The Annals of Mathematical Statistics 34 (963,

27 [GM07] [Le0] [Ne69] [NN94] C. Greenhill and B.D. McKay, Asymptotic enumeration of sparse nonnegative integer matrices with specified row and column sums, preprint arxiv: (2007. M. Ledoux, The Concentration of Measure Phenomenon, Mathematical Surveys and Monographs, 89, American Mathematical Society, Providence, RI, 200. P.E. O Neil, Asymptotics and random matrices with row-sum and column-sum restrictions, Bull. Amer. Math. Soc. 75 (969, Yu. Nesterov and A. Nemirovskii, Interior-Point Polynomial Algorithms in Convex Programming, SIAM Studies in Applied Mathematics, 3, Society for Industrial and Applied Mathematics (SIAM, Philadelphia, PA, 994. Department of Mathematics, University of Michigan, Ann Arbor, MI , USA address: barvinok@umich.edu 27