An upper bound on l q norms of noisy functions

Transcription

1 Electronic Collouium on Computational Complexity, Report No An upper bound on l norms of noisy functions Alex Samorodnitsky Abstract Let T ɛ, 0 ɛ /, be the noise operator acting on functions on the boolean cube {0, } n. Let f be a nonnegative function on {0, } n and let. We upper bound the l norm of T ɛ f by the average l norm of conditional expectations of f, given sets of roughly ɛ r n variables, where r is an explicitly defined function of. We describe some applications for error-correcting codes and for matroids. In particular, we derive an upper bound on the weight distribution of duals of BEC-capacity achieving binary linear codes. This improves the known bounds on the linear-weight components of the weight distribution of constant rate binary Reed-Muller codes for almost all rates. Introduction This paper considers the well-known problem of uantifying the decrease in the l norm of a function on the boolean cube when this function is acted on by the noise operator. Given a noise parameter 0 ɛ /, the noise operator T ɛ acts on functions on the boolean cube as follows: for f : {0, } n R, T ɛ f at a point x is the expected value of f at y, where y is ɛ-correlated with x. That is, y is a random binary vector whose i th coordinate is x i with probability ɛ and x i with probability ɛ, independently for different coordinates. Namely, T ɛ f x = y {0,} n ɛ y x ɛ n y x fy, where denotes the Hamming distance. We will write f ɛ for T ɛ f, for brevity. Note that f ɛ is a convex combination of shifted copies of f. Hence, the noise operator decreases norms. An effective way to uantify this decrease for l norms is given by the hypercontractive ineuality [3, 7, ] see also [0]: f ɛ f + ɛ. Entropy provides another example of a convex homogeneous functional on nonnegative functions on the boolean cube. For a nonnegative function f let the entropy of f be given by Entf = E f log f E f log E f where the expectation is taken w.r.t. the uniform measure on {0, } n. The entropy of f is closely related to Shannon s entropy of the corresponding distribution f/σf on {0, } n, and similarly the entropy of f ɛ is related to Shannon s entropy of the output of a binary symmetric channel with error probability ɛ on input distributed according to f/σf. The decrease in entropy or, correspondingly the increase in Shannon s entropy after ISSN

2 noise is uantified in the Mrs. Gerber s Lemma [5]: Ent f ɛ n E f φ Entf n E f,, ɛ where φ is an explicitly given function on [0, ] [0, /]. There is a well-known connection between the l norms of a nonnegative function f and its entropy see e.g., [6]: Assume, as we may by homogeneity, that E f =. Then Entf = lim log f. The uantity Ent f = log f is known as the th Renyi entropy of f []. The starting point for this paper is an alternative way to uantify the decrease in entropy after noise given in [3]. To state this result, let us introduce some notation. For 0 λ, let T λ denote a random subset T of [n] in which each element is chosen independently with probability λ. Let Ef T be the conditional expectation of f with respect to T, that is a function on {0, } n defined by Ef T x = E y:y T =x T fx. Then [3]: Ent f ɛ E T ɛ Ent Ef T. In light of the connection between entropy and l norms, it is natural to ask whether can be extended to an ineuality between l norms euivalently, Renyi entropies. The main result of this paper is a positive answer to this uestion. Theorem.: Let f be a nonnegative function on {0, } n. Then, for any > holds log f ɛ E T λ log Ef T, 3 with λ = λ, ɛ = ɛ r, where the exponent r = r is given by the following expression: { 3 r = ln < 4 ln Note that 3 is in fact an extension of, since assuming E f =, dividing both sides by and taking the limit as, recovers. The only thing to observe is that lim r =. Remark.: Since conditional expectation of a function f is a convex combination of shifted copies of f, its norm is smaller than that of f, and hence 3 measures the decrease in l -norm under noise, similarly to. In fact, the proof of 3 follows the approach of [7] to the proof of. In this approach, we view both sides of the corresponding ineuality as functions of ɛ for a fixed and compare the derivatives. Since noise operators form a semigroup it suffices to compare the derivatives at zero, and this is done via an appropriate logarithmic Sobolev ineuality see 7.. Applications Let C {0, } n be a binary linear code. Let r C denote the rank function of the binary matroid defined by C. That is, r C T is the rank of the column submatrix of a generating matrix of C which contains columns indexed by T. Applying 3 to the scaled characteristic function of C, that is f = n C C, gives the following claim, connecting between the linear structure of C and its behavior under noise.

3 Proposition.3: For any 0 λ holds λn E T λ r C T max << log E f ɛ λ/r, for ɛ =. 5 Let us discuss the bound given by 5. First, it seems useful to extend it to the limiting values of as well. Let F λ, = log E f ɛ, for < <. Set F λ, = lim F λ, = Ent f λ and F λ, = lim F λ, = log f λ ln. Then we get λn E T λ r C T max F λ,. For any λ, the three values of F λ, which seem to be easier to understand are F λ,, F λ, and F λ, = log E f. While simple examples λ ln show that F λ, can be both smaller and greater than F λ,, intriguingly we always have F λ, = F λ,, and moreover this uantity is easily expressible in terms of the weight distribution of the dual code C. Lemma.4: Let b 0,..., b n be the weight distribution of C. That is b i = { x C, x = i }, for i = 0,..., n. Then F λ, = F λ, = log b i λ ln i. i=0 We proceed to discuss some applications of 5... Codes achieving capacity for the binary erasure channel This is a family of codes for which we can effectively upper bound the LHS of 5, for λ close to the rate of the code. Recall that the rate of a linear code RC is defined as n log C = dimc n. A code C {0, } n more precisely a family of codes indexed by n achieves capacity for the BEC binary erasure channel if there exist two functions δ, δ which go to zero with n such that for λ = RC + δ the probability of decoding error given by p e = Pr T λ {r C T < RC n} is upperbounded by δ. This immediately implies writing R for RC that λn E T λ r C T λn δ Rn = δ n + δ Rn = on. Hence we have the following corollary of 5. Corollary.5: Let C be a BEC capacity achieving code of rate R. Then there exists λ = R + o such that for all holds F λ, on. The ineuality Ent f λ = F λ, on has been interpreted [], using as indicating that C is well-dispersed in {0, } n in the sense that the output of the binary symmetric channel with error probability ɛ = λ R whose input is a random codeword from C 3

4 is very close in the Kullback-Leibler distance to the uniform distribution. We observe that Corollary.5 can be interpreted as providing increasingly stronger measures of proximity of the channel output to uniform, as noise increases beyond R. The ineuality F λ, on together with Lemma.4 provide the following bound on the components of the weight distribution of a code C which is the dual of a code achieving BEC capacity. Proposition.6: Let C be the dual of a linear code C achieving BEC capacity. Let R = R C be the rate of C. Let b 0,..., b n be the distance distribution of C. Then for all 0 i n holds ln i b i on R In particular, since the dual of a Reed-Muller code is a Reed-Muller code, and since Reed-Muller codes achieve BEC capacity [9], the bound in Proposition.6 holds for Reed-Muller codes. The bounds in the literature [], [8], [4] seem to focus mostly on Reed-Muller codes of rates close to 0 or, or on weights which increase sublinearly in n, but do extend to all weights and to all rates. Comparing with these bounds, it seems that Proposition.6 improves the bounds on {b i }, for i growing linearly in n, for constant-rate Reed-Muller codes of almost all rates: 0 < R < δ, for δ δ < Bounds on weight distribution of linear codes The following more general claim is an immediate corollary of the ineuality λn E T λ r C T F λ, and of Lemma.4. Corollary.7: Let C be a binary linear code and let b 0,..., b n be the distance distribution of the dual code C. Then for any 0 λ and for any 0 i n holds b i λ ln i λn E T λ r C T. 6 This claim gives upper bounds on the components of the weight distribution of C, provided we can upper bound λn E T λ r C T...3 Ranks of random subsets in a binary matroid We give a different way to write the ineuality λn E T λ r C T F λ,, stating it as a lemma since it reuires a simple argument. Lemma.8: Let r C be the rank function of the binary matroid on {,..., n} defined by a generating matrix of a linear code C of length n. Let 0 p and let t = p ln. Then log E S r CS E T rc T. S p T t 4

5 Let the dimension of C be k. Recall that the Tutte polynomial [5] of the matroid defined by C is T C x, y = S {,...,n} x k rcs y S rcs. It is easy to see that an alternative way to write the ineuality of Lemma.8 is p k p n k T C p, + p p t k+ t n k d dy T C t,. t As an immediate implication of Lemma.8 we get the following tail bound. Corollary.9: For any 0 holds P r S p { S r C S E T r C T T t } +. Remark.0: Consider a different approach to obtain tail bounds for the function S r C S. Let fs = S r C S. Then fx fy x y, for any x, y {0, } n here x y stands for the Hamming distance between x and y and hence, by the bounded differences ineuality [4], for any t 0 holds P r S p {fs E T p ft + t} e t /n. Let µ : p E S p fs. By the Margulis-Russo formula [0], µ = p E S p i S fs f S \ {i}. Hence, since f is monotone increasing, µ 0 and µ is increasing. Moreover, since f is supermodular by submodularity of rank, it is easy to see that µ is convex. In particular, since µ0 = 0, we have µt t p µp for t p. Taking everything into account, we have P r S p {fs E T t ft + } e µt µp+ n e t pµp+p p n. Note that this does not seem to recover the bound of Corollary.9 when µt is small, as in the case of BEC capacity achieving codes. To conclude, we consider the special case of graphic matroids. For a graph G = V, E with n edges, let M be the matroid on {,..., n} whose independent sets are forests in G. This is a binary matroid, and Lemma.8 specializes as follows. Corollary.: Let G = V, E be a graph. For a subset of edges S E, let cs denote the number of the connected components in the subgraph V, S. Then log E S +cs t E + E ct. S p T t This paper is organized as follows. We prove Theorem. in Section. All the remaining proofs are in Section 3. Proof of Theorem. We start with a log-sobolev-type ineuality. Recall that the Dirichlet form Ef, g for functions f and g on the boolean cube is defined by Ef, g = E x y x fx fy gx gy. Here y x means that x and y differ in precisely one coordinate. 5

6 Theorem.: Let f be a nonnegative function on {0, } n. Then for any holds E f, f 4r E f n ln f ln Ef T, 7 where r is given in 4. T =n Moreover for < < euality is attained if and only if g is a constant function, and for if and only if there is a collection of subcubes of distance at least from each other such that f is constant on each such subcube. Corollary.: Let δ > 0 be arbitrarily small. For any nonnegative non-constant function f on {0, } n and for any > holds E f, f > 4 r δ E f n ln f ln Ef T. T =n Theorem. is proved below. For now we assume that it holds and proceed with the proof of Theorem.. Observe that the claim of the theorem is immediate for constant functions, since both sides of 3 are loge f. So we may and will assume that f is not constant note that this implies f ɛ is non-constant for all 0 ɛ < /. Fix >. Let δ > 0. We will prove 3 for λ, ɛ = ɛ r δ. Taking δ to zero will then imply 3 for λ = ɛ r as well. The proof proceeds by induction on n. The claim clearly holds for n = 0. Let n > 0. We assume that the claim holds for all dimensions smaller than n, and show that it holds for n as well. For fixed and δ both sides of 3 are functions of f and ɛ. Let F f, ɛ denote the LHS and Gf, ɛ the RHS. Clearly F f, 0 = Gf, 0 = log f. We will argue that if F f, ɛ = Gf, ɛ for some 0 ɛ < / then necessarily F f, ɛ < G f, ɛ here and below the derivatives are taken w.r.t. ɛ. It is easy to see that this implies F f, ɛ Gf, ɛ, for all 0 ɛ /, which is precisely what we need to show. Due to the fact that the noise operators form a semigroup under composition: T ρ T ɛ = T ɛ+ρ ɛρ, it will suffice to compare the derivatives at zero. This comparison is done in the following lemma. Lemma.3: Let g be a nonnegative non-constant function on {0, } n with E g =. Then F g, 0 < G g, 0. Proof: As shown in [7], we have F f, 0 = Ef,f E f. We claim that G f, 0 = r δ n log f T =n log Ef T, 6

7 which will conclude the proof of the lemma by Corollary.. In fact, note that λ0 = and that λ 0 = r δ. Hence we have: G f, 0 = d E dɛ log Ef T = d λ n log f + λ n λ log Ef T = ɛ=0 T λ dɛ ɛ=0 r δ n log f log Ef T, completing the proof. T =n T =n We can now conclude the proof of Theorem.. Assume that F f, ɛ = Gf, ɛ, for some 0 ɛ < /. Consider the functions F f ɛ, ρ and G f ɛ, ρ, for 0 ρ /. We have and F f ɛ, ρ = log f ɛ ρ = log f ɛ+ ɛ ρ = F f, ɛ + ɛ ρ, G f ɛ, ρ = E R λ,ρ log Ef ɛ R E R λ,ρ E log Ef T = T R,T λ,ɛ E log Ef T = E log Ef T = G f, ɛ + ɛ ρ. T λ,ρ λ,ɛ T λ,ɛ+ ɛ ρ The ineuality follows from the induction hypothesis for R [n] and from the assumption F f, ɛ = Gf, ɛ for R = [n]. Hence, by Lemma.3, F f, ɛ = concluding the proof. ɛ F f ɛ, 0 < ɛ G f ɛ, 0 G f, ɛ,. Proof of Theorem. We start with the base case n =. This case is dealt with in the following claim. Proposition.4: Let g be a nonnegative function on a -point space with E g =. Then E g, g 4r E g log g, 8 where r is given in 4. Moreover for < < euality is attained if and only if g is a constant function, and for if and only if g is a constant function, or a mutiple of a characteristic function of a point. 7

8 Proposition.4 is proved below. Here we assume its validity and proceed with the proof of Theorem.. First, we restate the -point ineuality 8 in an euivalent form, with two modifications: replacing g with E g and extending the ineuality by homogeneity to general nonnegative functions. Let g be a nonnegative function on a -point space. Then E g, g 4r E g ln E g E g. We can now extend this ineuality to general n by the following standard argument. Let f be a nonnegative function on {0, } n. Then, denoting by e i the i th unit vector, we have E f, f = E f x f x + e i fx fx + e i. x i= For x and i, let f x,i denote the restriction of f to the -point space x, x + e i. Then the RHS is n i= n x:x i =0 E f x,i, f x,i. By the -point ineuality, this is at least 4r Let θ x,i = i= n E f x,i y:y i =0 E f y,i x:x i =0 E f x.i ln E f x.i E f x,i = 4r i= n x:x i =0 E f x.i lne f x,i E f x.i. Then x:x i =0 θ x,i =. Note also that y:y i =0 E f y,i = n E f. By convexity of ln the RHS above is at least 4r E f E f x,i ln θ x,i x E f x,i = 4r E x f proving 7. ln i= i= x:x i =0 E f E Ef [n] \ {i} = 4r E x f x:x ln i =0 E f x,i n E x f i= = 4r E f n ln f T =n ln Ef T, It remains to consider the cases of euality. Tracking back the conditions for euality in the proof above, it is easy to see that euality holds for f if and only if it holds for all one-dimensional restrictions f x,i. For < < this means that all these restrictions are constant functions, implying f is constant. For this means these restrictions are either constant, or multiples of a characteristic function of a point. It is easy to see e.g., by induction on dimension, that this implies that there is a collection of subcubes of distance at least from each other such that f is constant on each such subcube.. Proof of Proposition.4 There are two cases to consider, < < and. These cases are dealt with in the following subsections, starting with the somewhat easier case. 8

9 .. The case We will show that for a nonnegative function g on a -point space with E g = holds E g, g ln E g ln E g. 9 Observe that euality holds in two cases: if g is a constant- function, a multiple of a characteristic function of a point. We will show that these are the only two cases for which euality holds. Assume, w.l.o.g., that f0 f. Let 0 x, and let f0 = x and f = x. Then 9 becomes: x x x ln x + x x + x ln Setting y = x x + x x, z = x + x, and rearranging, this is the same as y z log z. Note that since x = 0 is one of the cases for which euality holds, we may assume that x > 0. Now, let t = x x. Then t, y = t +t t+, and z = t + t+. Rearranging and simpifying, 9 becomes t + log t + t + t t, for t,. + By convexity of the function a a, we have < r = t+ t +. By concavity of logarithm log r r. We remark that the only case in which euality holds in this ineuality is if r =, that is t =, and this implies g is constant. Hence, it suffices to prove t+ t + t+t + t +. Simplifying and rearranging, this is the same as t + t + t +. Let h be a function on the two-point space with values t and. Then the ineuality above is the same as h h h. Recall that the function F : p ln h /p is convex this is a conseuence of Hölder s ineuality. Rewriting the last ineuality in terms of F, we get F + F F, which is true by the convexity of F. This completes the proof of 9. Tracing back the conditions for euality in 9, it is easy to see that it holds only in the two cases mentioned above. 9

10 .. The case < < We will show that E f, f ln r E f ln E f, and that euality holds iff g is a constant- function. r = 3, 0 Let us first deal with the case g0 = 0. In this case, the ineuality reduces to 3 = 4 3, and it is easy to verify that this is a strict ineuality for < <. Proceeding as in the proof of the first case, using the same notation, and writing r for r, the ineuality 0 is euivalent to y z r log z x. Substituting t = x note that we may assume x = g0 > 0 and rearranging, this is the same as t + r log t t + t + t + r. + Since < t+ t + Euality holds only if t+ t + would suffice to show that r r t+ t +, concavity of the logarithm implies log t+ t + t+t t + + r +. Multiplying both sides by t + gives t+ t +. =, that is t =, which means that g is constant. So, it t+ t + t+t t + + r, which is the same as r t + t + t + r + t +. Writing the above as F t Gt + Ht, it is easy to see that F = G + H and that F = G + H. So it suffices to show F t G t + H t, which amounts to after some simplification and rearranging: or r t + + t 3 + r t t + 3 r t r t t t 3 + t, + t. Substituting y = t+ t and rearranging, this is the same as proving for < y that r y + y r +. Denote the LHS by hy. Then h is proportional to r y 3. Substituting r = r = 3, we see that h is proportional to y 3. Hence h < 0 for all y <, and it suffices to verify the last ineuality for y =. Substituting y = we get an identity in. This completes the proof of 0. Tracing back the conditions for euality, it is easy to see that it holds only if g is a constant function. 0

11 3 Remaining proofs Proof of Proposition.3 We will need some facts from Fourier analysis on the boolean cube [0]. First, recall that for a function f on {0, } n and for a subset T [n] holds E f T = frw R T R, where {W R } are the Walsh-Fourier characters. Recall also that if f = n C C, where C is a linear code, then f = C. We also recall a fact from linear algebra. For T [n], let CT be the subspace { R T, R C }. It is well known and easy to see that dim CT = T r CT, where r C is the dimension of the subset of columns indexed by T in a generating matrix of C. Hence, for x {0, } n : E f T x = { C w R x = T = T r CT if x CT 0 otherwise R C T And hence, E f T = E E f T = T r CT. Using this in 3 gives, for any > that log E fɛ E T r C T = λn E r C T, T λ T λ where λ = ɛ r. Rearranging, we get the claim of the proposition. Proof or Lemma.4 Recall that for a function f on {0, } n and for 0 ɛ / holds f ɛ R = fr ɛ R. Let b 0,..., b n be the distance distribution of C. Then, since ɛ = λln, we have F λ, = R {0,} n f R ɛ R = i=0 ln i b i λ On the other hand, it is easy to see that for f = n C C and for any 0 ɛ / holds f ɛ = fr R {0,} n ɛ R w R = fr R {0,} n ɛ R = f ɛ 0 since f is nonnegative. Hence, recalling ɛ = F λ, = f ɛ = f ɛ 0 = λ ln, R {0,} n fr ɛ R = b i λ ln i. i=0

12 Proof or Lemma.8 Let ɛ = p. Using 3 with = we have, for f = n C C, that E f ɛ = S {0,} n f S ɛ S = S {0,} n f Sp S = E S p E Ef S = E S rcs. S p Hence the claim of the lemma follows from the ineuality λn E T λ r C T F λ, with λ = ɛ r = ɛ ln = p ln. Proof or Corollary. For S E, the matroid rank of S is given by rs = cs i= C i = V cs, where {C i } are the connected components of V, S. Hence, S rs = S + cs V. Substituting this into the claim of Lemma.8 gives the claim of this corollary. Acknowledgments I am are grateful to Or Ordentlich for many very helpful conversations and valuable remarks. References [] E. Abbe, A. Shplika, and A. Wigderson, Reed-Muller Codes for Random Erasures and Errors, IEEE Trans. Information Theory 60: 59-55, 05. [] W. Beckner, Ineualities in Fourier Analysis, Annals of Math., 0975, pp [3] A. Bonami, Etude des coefficients Fourier des fonctions de LpG, Annales de linstitut Fourier, 0 970, [4] S. Boucheron, G. Lugosi, and P. Massart, Concentration ineualities, Oxford University Press, 03. [5] T. H. Brylawski and J. G. Oxley, The Tutte polynomial and its applications, in Matroid Applications, Encyclopedia Math. Appl. 40, N. White, ed., Cambridge University Press, 99. [6] T. Cover and J. Thomas, Elements of Information Theory, Wiley 006. [7] L. Gross, Logarithmic Sobolev ineualities, Amer. J. of Math., 97, 975, pp [8] S. Kudekar, S. Kumar, M. Mondelli, H. D. Pfister, and R. L. Urbanke, Comparing the bit- MAP and block-map decoding thresholds of Reed-Muller codes on BMS channels, ISIT 06.

13 [9] S. Kudekar, S. Kumar, M. Mondelli, H. D. Pfister, E. Sasoglu and R. L. Urbanke, Reed- Muller Codes Achieve Capacity on Erasure Channels, IEEE Trans. Information Theory, 637, [0] R. O Donnel, Analysis of Boolean functions, Cambridge University Press, 04. [] O. Ordentlich, Y. Polyanskiy, and R. Urbanke, personal communication, 06. [] A. Renyi, On the measures of entropy and information, in Proc. 4th Berkeley Symp. on Math. Statist. Probability, Berkeley CA 96, vol., [3] A. Samorodnitsky, On the entropy of a noisy function, IEEE Trans. Information Theory 60: [4] A Shplika and O. Sberlo, Weight Distribution of Reed-Muller Codes with Linear Degrees, manuscript 08. See also O. Sberlo, MSc thesis, WIS 08. [5] A. D. Wyner and J. Ziv, A theorem on the entropy of certain binary seuences and applications: Part I, IEEE Trans. Inform. Theory, vol. 9, no. 6, pp , ECCC ISSN